One big issue with all those NN-generated texts, be it GPT3 or this, is that it's just a collection of seemingly random words.
I think that the issue lies in that we approach content generation in the wrong way: a sequence that goes from start to end in one go. The idea is at the start and by the end of the sequence, the model had moved far away from the original idea. We humans, start with one or a couple of ideas and we iterate over them many times and then expand them in all directions. If only a model could do that.
I think the real problem with GPT3 etc is it's not really original.
It's a corpus of text so large and varied, that simple requests can bring up a few good examples, which can be mashed together to seem new. It's essentially a smart search & plagiarism process. I suspect if you could view the main contributing sources it would be less impressive.
for example, I was really impressed that GPT3 could write code - until I realised that it contained most of stack-overflow (including code examples) - no actual inference was happening, it was just search.
That's not true, or is at least making GPT3 out to sound more trivial than it is. I've spent hours "interviewing" people about dragon sightings in Washington D.C., or Elon Musk about a fictional startup, or similar. I've had it generate startup ideas that are definitely original and elaborate on them, including company names that are concatenations of words that have never been used as a proper noun. I had it generate an interview with the founder of a fictional 1980s video game company, and it went into detail about the series of fictional games. It was all plausible, but the names were unique strings that I could not find on Google.
I'm not impressed by fictional nouns, be they people names or fictional corporations. GPTs special sauce was supposed to be text by, at the least, the paragraph, so it's the "generate/elaborate on startup ideas" and "went into detail about the series of fictional games" I'd be interested in.
Without seeing the source material, how original do you know these generations to be?
I've had to explain this to non-techy family. GPT3 is still a monkey at a keyboard. Only, the keyboard is huge, and each button is a whole phrase. Maybe the keyboard reconfigures itself after each keypress, to make the phrases scan a bit better. But it's still a monkey at a keyboard.
On the one hand, there always seemed something kind of flawed about the work OpenAI is doing versus DeepMind.
On the other hand, people talking about OpenAI all the time helps it thrive - all publicity is good publicity. So the best thing you can do personally is, if you don't like what they do, just don't talk about it anymore.
It's a know technique, I wouldn't say that makes it "valid". It also seems to be a inspirational technique i.e. adding an artificial limit to spur creativity.
The text is ultimately managed by a real human being otherwise capable of writing original text. That human being hasn't been exposed to English entirely through the corpus of test given, as would be the case of GPT, which has no previous knowledge of language.
TLDR; the technique gives the text to an already create human being, rather than expecting creativity to arise from the technique itself.
When I was in high school, some friends and I would pass around a piece of paper and write a sentence or two of grammatically correct but largely nonsensically content. The result was an amusing, silly story, entertaining to us and perhaps my close circle of friends, but less so to outsiders. We called them "cheese logs" and later adapted the style on the web, back when personal web sites were "en vogue."
This is very similar to the parlor game “exquisite corpse”.
A fun modification is is provide the next writer the last word (or the last few words) to build from.
It is also fun to play as a drawing game. Divide paper into thirds. One player draws a set of shoulders/head leaving a few connection lines passing below the fold. Fold paper over so only the connection lines are visible. Next player uses those lines to draw the torso. Fold. Last player draws the legs.
It’s good to have a few going at one time so everyone gets to draw throughout.
We wrote exactly 5 words at a time and the results were shockingly sacreligious, violent, sexual, and nonsensical. Reading them aloud still makes me laugh to the point of hyperventilation.
> An algorithm (pronounced AL-go-rith-um) is a procedure or formula for solving a problem, based on conducting a sequence of specified actions
As far as I'm concerned all the cookie-cutter blockbuster movies of the past 10 years have been written according to an algorithm and they were all terrible.
I guessed that based solely on watching some of their trailers. Having a hard time remembering the last movie I saw, but I think it was Star Trek 2 and it was indeed terrible. Benedict Cumberbatch the typecast troubled aspie was almost, but not entirely the wrong choice for Khan.
I imagine these algorithm writing tools (GPT, etc) are going to be adopted by writers the same way autotune has been embraced by the music industry. When used in moderation it can help give originality to the final work. When overused it will sound like a parody.
Yeah I was surprised at first. Since I remember seeing the movie in a museum years ago. Watched it for 2 or 3 loops before reading the description explaining why any of the nonsense was happening :)
