The final two weeks leading up to the deadline were incredibly chaotic for the team working on the transformer model. While their official desks were located in Building 1945, they spent most of their time in Building 1965 due to the superior espresso machine available in the micro-kitchen. The team, led by Gomez, was in a constant state of debugging and experimentation, trying out different combinations of tricks and modules in an attempt to optimize their model. The process involved numerous ablations – removing certain elements to test the effectiveness of what remained. This highly iterative trial and error phase produced a “minimalistic” output that was essential for the success of the project.
During the intense work on the transformer model, Vaswani found himself struck by inspiration one night while crashing on an office couch. He noticed a pattern on the curtains that reminded him of synapses and neurons, leading him to realize the potential for uniting various modalities – speech, audio, and vision – under a single architecture. This vision of a more generalizable model indicated that the team was onto something groundbreaking beyond just machine translation.
Despite the tremendous potential of the transformer model, the higher-ups at Google initially viewed it as just another interesting AI project. The team members admitted that their bosses did not frequently inquire about updates on the progress of the project. However, they were aware of the significant impact their work could have and were driven by the desire to push the boundaries of transformer models beyond text and into other forms of human expression like images, audio, and video.
As the deadline approached, the team realized that they needed a catchy title for their paper. After a brief brainstorming session, they settled on the title “Attention Is All You Need,” inspired by a radical rejection of conventional best practices like LSTMs in favor of attention mechanisms. The decision to name their paper after a Beatles song was a spur-of-the-moment idea by Jones, a British team member. The team worked tirelessly until the very last minute, with Parmar even entering the final English-French numbers just five minutes before the submission deadline. Ultimately, they managed to send off the paper with only two minutes to spare.
The process of creating the transformer model was a testament to the team’s dedication, innovation, and adaptability. Despite facing intense pressure and working under tight deadlines, they were able to produce a groundbreaking model that has paved the way for further advancements in the field of deep learning and artificial intelligence. Their story serves as a reminder of the importance of perseverance and collaboration in pushing the boundaries of technological innovation.
Leave a Reply