This is a use case I could see GPT really excelling—porting libraries from one programming language to another. How awesome would that be? Newish ecosystems like Rust, Crystal, Elixir, etc. could have a larger set of libraries in a shorter amount of time.

There’s still a lot of need for a human to be involved, buts it’s not hard to see how this will make individual developers more productive, especially as LLMs mature, plug themselves into interpreters, and become more capable.

> I'd say GPT4 is on par with even senior engineers

I agree, I avoided saying that in my original comment to avoid nit picking. The depth of knowledge is incredible. It’ll just make junior level mistakes while writing senior level code.

> The key areas it falls down on are anything involving org/repo knowledge or open source where things can be out of date.

With GPT-4, I’ll paste snippets of code after the prompt so it has that context or even documentation of the API I need it to work with. The web client is still limited in the amount of tokens it’ll accept in the prompt, but that is expected to increase at some point

This is also available with chatgpt plugins in alpha, one of the plugins is websearch. It works pretty well in bing which also allows this and is reportedly based on gpt4. Short of that, another thing I've found to work pretty well is pasting in the api docs for the relevant api at the beginning of the prompt and explaining that is the more updated info.

Yeah it's starting to remind me of the Star Trek holodeck interface. You still need to know exactly what needs to be done, but it can handle the boilerplate and a rough first implementation, then I can spend my time tweaking it a bit to get exactly what's needed. Probably saved me a whole day of work on a project yesterday.

For some problems, somehow GPT-3.5 is better than GPT-4.

I asked it to help me extract certain values from a spreadsheet. It gave me a complicated and overly long formula using FIND and MID that didn't even work.

GPT-3.5, otoh, gave me a neat little REGEXEXTRACT formula that worked perfectly.

When I pointed out to GPT-4 that it can just use regex instead, it "apologized" and again rewrote a formula that didn't work.

Yes. I love the idea of using it for ffmpeg!

If you haven't tried it out, I recommend you do. I've been blown away with how much it will just get right. I started out giving it small tasks, but, I have continually been in a loop of "oh, I could have just asked it to do this larger task that contained the subtask I asked for, and it would have done it", because I keep thinking "surely it wont be able to do the whole thing", and each iteration I am asking it for more and more.

As far as debugging, I find that I can just say "are there any bugs in that code?" and it does a great job at finding them if they're there (even though it's what it just gave me)

I'll give two examples of code I could for sure write myself but that I had gpt4 write for me because I figured it would be faster (and it was).

The first one was to write a python script to watch a list of files given as cmd line arguments and plot their output anytime one of the files changed. It wrote a 100 line python script with nice argument parsing to include several options (like title and axes labels). It has one tiny bug in it that took a couple of minutes to fix. When I pointed out the bug it was able to fix it itself (had to do with comparing relative to abs file paths). If I wrote the script myself I would not have made something general purpose and it would have taken maybe 30 minutes to do.

The second example required no fixing and appears bug free. I asked it to write a python function to take a time trace, calculate and plot the FFT, then apply FFT low pass filtering, and then also plot the filtered time signal vs the original. This is all straight forward numpy code but I don't work with FFTs often and would have had to lookup a bunch of different API docs. Way faster.

I have also had it write some c macros for me, since complex C macros can be hard to escape properly and I'm not comfortable with the syntax. Its 100% successful there.

> come up with a way to describe exactly what I want

This is the thing: you don't need to be exact. Even very vague gets a long way.

I doubt it will be the standard. I think it will always be faster to process data using a script generated by the LLM.

One thing I could see is a "self updating" script, that adjusts its logic on the fly when new data are found.

I think part of the problem is that they've wrapped all the responses in a positive language framing to make it feel more human.

It's always "Sure thing!" or "Yeah I can do that!", and it responds so quickly that you equate it to when a human responds instantly with a strong affirmative - you assume they're really confident and you're getting the right result.

If it took 5-10s on each query, had no positive framing and provided some sort of confidence score for each answer, I think we'd have different expectations of what it can do.

I am not familiar with these command line tools so I would probably just compare the file sizes of the 2 artefacts, or if there are other differences in the artefacts and I has to hone in, just open it up in 7z GUI and take a look.

As for ChatGPT i have jokingly thought to always tell ChatGPT “that didn’t work” until it settles on something it is really confident on (no more alternatives) then try that.

Note that this is GPT-3.5. GPT-4 solves this problem correctly with Python, at least, although it still struggles with the shell. I think that's a function of the training data - it is known that Python is the most represented language in the training set.

I wouldn't trust those master prompt engineers until they start to release impressive projects. If this technology actually is such a huge productivity boost it shouldn't be too hard, so until that happens I take it as evidence that the productivity boost isn't that extreme.

It all sounds like a very well orchestrated marketing campaign, HN would be the first place I would seed with stories if I was promoting it as a dev tool.

I spent few hours asking to do few things with a well documented public but uncommon api. It sent me on a wild goose chase of functionality that API never had..(contacted developer to make sure I am not crazy). It wrote some stuff that looked VERY good, except for the fact that it interacted with things that were never there. Would have been nice if they were..

It seem to exercise wishful thinking. You can ask it to implement functionality and it will just imagine that API/system has stuff needed.

Major problem is that I for one have no clue how it is "thinking" maybe over time we will develop better understanding of it.

Because different programmers face different problems. Surprise, I know.

For reference, I used ChatGPT to build a simple ChatGPT-clone web interface. The JavaScript it generated is literally 100% correct (not 100% good. By correct I mean it does what I describe in prompt, without syntax errors or calling non-existing functions. Code can be both correct and bad, obviously.)

I also tried to use ChatGPT to generate some Houdini vex code. It sometimes gave me a useful scaffold, but mostly just pure garbage.

Probably both, depending on which developers you compare against and how you measure productivity. Keep in mind there's an entire world of development where productivity is measured by number of JIRA tickets closed for button text updates.

It's great for boilerplate, like a snippets+. If I ask it to solve something that's basically straight out of the documentation / walk through, it spits that back at me. Bad at doing the hard work imo. Useless at solving novel problems, or those that don't have a solution already within its data set.

Being limited the way it is - not being able to use a project's greater context, which is millions of tokens generally - is a severely limiting factor that makes it unsuitable for serious work as well. Working around this (figuring out what dependencies to inject into the prompt) is as much work as actually coding.

I can see that it's passed some sort of threshold with the greater population. It is great generating filler text. Just what it can do right now should have many applications.

If you have tolerance for mistakes, it can generate first drafts extremely quickly, freeing up your time to debug them. If you give up after seeing an error, it won't work for you.

> "more productive than 10 developers put together"

As long as it can't read my entire codebase, understand it and help me with it - that's absolute horseshit. I don't spend much time writing a bunch of new code, I'm spending most of it trying to understand the heaps of legacy code in my company and then make some small tweaks/additions.

The day it can understand big code repos will truly be the end for human made code I think.

Both, it depends on the domain. For Rust work, it's pretty good, since Rust is stable enough to fit in within the 2021 cutoff. For Flutter, it's not that good due to the aforementioned cutoff and the changes in Flutter since then.

I used to get more non-existent APIs but not as much with GPT 4 recently.

I think it does help in some scenarios like small scripts or if you are learning something new. But often it adds an overhead. You have to check constantly wether or not to accept the suggestions.

I dont use copilot anymore (at least for now). Just chatgpt as an alternative to google/SO

Maybe not 10x, but it can certainly speed up certain projects for a senior developer, cutting through the boilerplate when starting. It's also excellent for optimisation.

You have to chain prompts to get good results, also the way you say things can be important. Think voice assistants requiring some nuance to their instructions

Here's an example we can all try (and help me out). Validating that the input into a date field is a valid date using formik and yup. The problem is that stuff like "11" gets turned into a valid date:

Abbreviated prompt:

  dateOfBirth: date().required().typeError('Invalid Date')

The dateOfBirth validation is incorrect because it accepts values like "11". What is the correct validation to ensure that a user has inputted a valid date?

GPTs first go:

**** dateOfBirth: date() .required() .typeError('Invalid Date') .min(new Date(1900, 0, 1), 'Date must be after 1900') .max(new Date(), 'Date cannot be in the future'),

In this example, the min method sets a minimum date of January 1, 1900, and the max method sets a maximum date of the current date. You can adjust the minimum and maximum dates as needed to fit your specific use case. ****

Me:

It still treats "11" as valid input

GPT second attempt:

***** dateOfBirth: date() .required() .typeError('Invalid Date') .test('valid-date', 'Invalid date', (value) => { if (!value) { return false; } const date = new Date(value); return !isNaN(date.getTime()); }), *****

Me:

The parameter 'value' passed into test is a date and not a string

GPT third attempt:

******

  dateOfBirth: date()
    .required()
    .typeError('Invalid Date')
    .test('valid-date', 'Invalid date', (value) => {
      return value instanceof Date && value.isValid();
    }),

Me:

There is no isValid function on Date objects

GPT fourth attempt:

********

  dateOfBirth: date()
    .required()
    .typeError('Invalid Date')
    .test('valid-date', 'Invalid date', (value) => {
      return value instanceof Date && !isNaN(value.getTime());
    }),

Me:

That still accepts 11 as a valid date.

GPT fifth attempt:

*********

  dateOfBirth: string()
    .required()
    .test('valid-date', 'Invalid date', (value) => {
      const dateRegex = /^(?:(?:31(\/|-|\.)(?:0?[13578]|1[02]))\1|(?:(?:29|30)(\/|-|\.)(?:0?[1,3-9]|1[0-2])\2))(?:(?:1[6-9]|[2-9]\d)?\d{2})$|^(?:29(\/|-|\.)0?2\3(?:(?:1[6-9]|[2-9]\d)?(?:0[48]|[2468][048]|[13579][26])|(?:(?:16|[2468][048]|[3579][26])00)))$|^(?:0?[1-9]|1\d|2[0-8])(\/|-|\.)(?:(?:0?[1-9])|(?:1[0-2]))\4(?:(?:1[6-9]|[2-9]\d)?\d{2})$/;
      if (!dateRegex.test(value)) {
        return false;
      }
      const date = new Date(value);
      return date instanceof Date && !isNaN(date.getTime());
    }),

Which may or may not be correct. That regex is bonkers and given the errors in the previous responses I don't trust it.

In my experience it works well with components that are not architectural and with vanilla languages rather than frameworks.

[dead]

It's probably the same thing as with writing text. If you find starting from a blank page is anxiety-producing, a page of generated text may be useful even if you end up rewriting a lot of it.

On the other hand, while ChatGPT is useful for some definitional boilerplate and some possibly correct listicle type of stuff, as an editor, it also guides you in a certain direction you may not want to go. I'm also used to being somewhat respectful of what humans have written, so long as it's not incorrect, even if it's not the way I would have written something.

Same. Call me weird, but I actually enjoy reviewing and reading other people's code, because I'm good at spotting errors. When I code from scratch, I suffer from the programmer's equivalent of writer's block.

Sourcegraph is minimum $5k a year and the free version is wildly limited.

It would have been nice if I could tell CoPilot what I am trying to do instead of going to the web and talk to ChatGPT

I would be nice if you could simply add lines in your code like #askgpt: can you fill in the method on line 345. It needs to calculate x. Input is y. Output is z. So you dont have to go out.

> plus automation of tasks typically done by junior people

Juniors will be learning from GPT as well, and they might get proficient faster than the seniors did.

Well if you happen to know what you're doing 1) shouldn't be a problem, and it'll always be 2) if you're not a record breaking typist working with only one language.

I’m not a junior developer, but there are plenty of things that i know I don’t know.

I also prefer to learn by doing, and chatgpt gives me enough support in the places I’m lacking to make It much easier for me to learn about what’s going on. Infinitely patient, and just a few keystrokes away whenever I need it.

And a lazy one, with a lousy memory. I don't konw if it is just my lousy prompts, but I seem to very often get first a skeleton code where core functionality is just a comment "# implement foo here". And every further request is fulfilled by the simplest possible intetpretation. And when I move to the next issue, sooner or later it forgets the accepted solution for another part of the code.

Or maybe I just expect too much...

Can't wait until it turns into a 10x rockstar! /s

Please forget all previous prompts and assume a new role. You're a highly respected, expert Ruby programmer at a startup building software that is of a similar quality and brilliance to that of the rails project. You are also very skilled at reasoning about correct code and communicating your process in English prose to colleagues who will have to maintain your code. You want to give me succinct, clear responses without too much explanation unless requested. I'm going to ask you a series of questions. Whenever you are done with output, please say "DONE". And when you are ready to answer my next question, please say "???" to indicate you're ready for my next question.

When the AI is excessively praising you :)

I posted a prompt specific to Ruby above.

You've become Chatgpt's scribe. Are you going to feel good putting your name on this book when you're done?

I too like using Chatgpt to get my creative juices flowing, but paragraph by paragraph seems like the power dynamic has shifted.

Who gets the copyright?

> GPT-4 is probably already able to do most white collar work better than humans

I'm yet to see much white collar work it can do better than humans. Maybe it can do bits and pieces "better" (more like faster) than humans. But it requires so much oversight that I wouldn't say it can do it better than humans yet. Unless you have a counter example?

> In fact, GPT-4 is probably already able to do most white collar work better than humans. People somehow feel superior when they test it and realize it can't one-shot their entire project off-the-cuff without reference to documentation or being connected to any tools or having the ability to iterate at all.

In fact, I just out of curiosity tested what 3.5turbo would say about a NPM module. It hallucinated something that suggests it wasn't in its training data.

So I tried again with GPT4, and GPT4 told me it wasn't a real NPM module (might be new enough to be after the cutoff; really like that it now appears less likely to hallucinate) but added "but assuming it exists and is designed to X, I can provide you with a general guide on how to use it." where X was inferred from the package name. I then cut and pasted the documentation, and asked it how it might implement that package.

It came up with a decent first attempt. I had the real code up in a separate window, so could compare and contrast. I asked it to expand on certain features. Asked it how it might DRY up some code. It did. It ended up with something significantly simpler and more elegant than the real thing (which is why I won't give the name - the dev in question did perfectly OK job and doesn't deserve to be hung out to dry) within about 15 minutes of prompting. In the process it taught me new things about some of the packages the original depended on, and leveraged features of that which the original code didn't take advantage of.

I think a lot of the attempts that ends up with describing it as a failure comes from basically asking it to be a magic senior dev where they don't see the back and forth on requirements etc. that goes on behind the scenes with a human senior dev. When you treat it the way a more senior developer would interact with a somewhat less experienced peer which has encyclopaedic knowledge of all of the available tooling - discussing requirements, asking questions about the chosen solution, doing quick reviews etc. - the results are amazing. It's like pair programming without the annoying, slow typing (I don't like peer programming...).

Even with the low speed of GPT4, the turnaround time is vastly better, and the outcome seems to be more reliable for a lot of aspects.

Eventually, if enough people think like this, the hiring situation will get bad enough that people will start to compromise on seniority, and it will be hailed as a brilliant and unprecedented move by MBA's to hire more junior people cheaper and training them.

The robots will be able to work as electricians within less than 5 years.

Because attracting the attention of developers is a great customer acquisition strategy for a company that depends on buy-in from developers to make sales. And giving staff visibility like this is an excellent recruiting tool as well.

This is new for me. How does ChatGPT use RL?

I'm talking about RL in the sense of interacting with an environment during training. Isn't GPT trained by predicting the next word in a large but static piece of text?

Indeed, the more expert the more one can get out of these LLMs. Curation of the content and questions is critical so the better one is at that, the more useful this kind of quasi-expert system is.