So in my case, the Finanzamt argues that what I did for the startup does not fulfill the definition of a "gewerbliche Tätigkeit" and consequently I can't deduct the "Vorsteuer" of my expenses. (As a business you get the VAT you paid on expenses back from the Finanzamt, which is called Vorsteuer.)
Download the Umsatzsteueranwendungserlass and go to page 63 of that PDF (section 2.3. "Gewerbliche oder berufliche Tätigkeit") to find a list of things that indicate whether you're having a business or not:
In theory, if you fulfill the definition in that PDF, you're on the safe side. In practice however, the Finanzamt often tries to fleece you by simply making a claim that something can't be deducted, shifting the burden of proof to you. My particular case is still in dispute, I have an appeal (Einspruch) running against their decision. Most likely they'll lose, but they try to fleece people nevertheless because they figure that people will just pay and not spare the time to counter their evil tricks. If you do decide to shoot back, you have to be a German native speaker so you can read the laws and you should have some experience with the tax system. Alternatively, get a Steuerberater but be prepared to waste more money on your Steuerberater than what you would otherwise have paid for the Finanzamt's tricks.
One final word of warning: If your startup fails and you wind down the company, be prepared that the Finanzamt will conduct a detailed examination of the tax declaration of the past 3 years (Betriebsprüfung/Außenprüfung). This can drag on for several years because they're pretty slow. Apparently they routinely do this whenever a company is shut down.
