For my use case, overall Marker seems to work pretty well - but it has issues with tables. Merged cells, misplaced headers, and so forth. I'm currently extracting Polish PDFs that are //not// scanned
When compared to Azure Document Intelligence, Marker is really cheap when self-hosted (assuming you fall under the license requirements), but it does not produce high quality data. YMMV.
This is why we created Lun. Every gas boiler's consumption emits about 2tCO2 per year - and there are about 90 million buildings in Europe that could be heated using heat pumps instead, with huge benefits for energy efficiency and carbon emissions.
As you said, it's a hard problem.
Luckily, politicians and bureaucrats are passing more and more legislation that will force the fossil fuels out of heating over the next decade. We see it as our job to make sure it's going to happen in real life by being the glue between all parties in the value chain (consumers, OEMs, installers, utilities, governments, financial institutions).
And we're looking for great people ;) careers.lun.energy
For where I live in London, ground heat pumps are infeasible and air heat pumps would produce too much noise and not be permitted under conservation area rules.
What most of the UK needs is an effective drop-in replacement to a combi boiler, ideally one that doesn't need a large water storage cylinder. It could even be a combination of a slow boiler (for radiators) and a fast water heating unit (for bath/shower/taps).
Any retrofit of air or ground heat pumps into Victorian terraces just wouldn't work or be acceptable.
As yet no comparable electric version exists at similar price, size, capability. This is where a govt can incentivise innovation to both create what we need, and then help it be deployed (first with a stick like banning things, and second with a carrot like helping those in poverty get it for cheap/free).
Requires land for the district heating facility, and then ripping up the road to lay plumbing, etc.
For the UK the solution is known: create and mass produce an electric alternative to the combo boiler.
The numbers required, and the time to fit them offers at least a decade of mid-skilled manufacturing and trades employment, as well as recycling opportunities for the old boilers (aluminium, steel, brass).
The cost can also be spread over time and you can start anywhere once the boilers exist.
We're also super proud that the data is being used by Google to ensure that they use their data centers more when the electricity is greener and to help them in goal to make their electricity usage 24/7 carbon free in 2030 (https://www.gstatic.com/gumdrop/sustainability/247-carbon-fr...)
Great collab with Google! They've been probably the most serious corporate wrt getting all-renewable offsets for their operations... this is helping them reach the next milestone where the renewable offsets are time-matched. Great example of being serious about this stuff rather than just greenwashing.
It sounds like this project is for their own operations. Have you guys thought about how to offer this closer to a turnkey cloudops SASS / API? What kinds of abstractions would you present to developers building non-time-critical compute loads?
Could be a great differentiator for GCP vs. AWS (I have heard of some companies choosing GCP over AWS due to Google's green energy cloud). And for you guys, the only thing better than Google being a customer is all of Google's customers being your customers.
Also, how can we avoid the potential for unintended consequences where this tech makes Google "greener" while GCP users become less green?
If a data center has (roughly, to a first-order approximation) fixed compute capacity at any point in time, and we assume that any capacity not being used by Google themselves is made available to GCP, then wouldn't Google reserving the "green" hours for themselves drive the remaining "dirty" hours onto the GCP spot markets?
Is there a cloud market design that addresses this tension between maximizing utilization and having desirable or 'premium' compute hours?
That looks like a good opportunity for an (at least tangentinally related) big shout-out to your great electricityMap (https://www.electricitymap.org/map) website!
I wish there would be more countries covered (in particular Switzerland), but I guess you depend on the live data being provided in these countries.
Unfortunately, the crux is data availability and reliability from system/transmission operators. For example, there is no online data available for the Northern Territory of Australia, so you can't build a parser for it. Some data providers have frequent data outages for different regions (ENTSOE), as there is no SLA or contractual obligation for providing data reliably.
If you're aware of a region that has live data available and is not yet live on electricitymap.org, please consider contributing a parser [1]! If you live in a region without live data, please consider politely requesting such data be made available through utility and system operator contacts, or explore requiring such data be made available by law (if public policy is your thing).
I think the problem is that something is wrong with the ENTSO-E feed for solar and wind power in Switzerland. Data is given as "N/A" on their webpage, too, and it states that: "Due to changes to a different primary data provider, no data is published for Production Type Solar and Wind Onshore. The missing data will be published retrospectively by 28th Mar 2019."
Given that the 28th Mar 2019 has long passed, I am not very optimistic that the feed will be restored anytime soon... and unfortunately I don't know of any other (e.g. primary) public source, either.
Hey Martin, this is super cool! Congrats on the collab!
Would you mind commenting on what your tech stack is like? Looking at your github repo it seems like you're combining a lot of data sources. Can you comment on your approach?
Also, considering this collaboration, are you running on GCP?
Hello,
Olivier here (CEO Tomorrow)
Indeed we're running on GCP, using a mixture of Python and Node. In terms of the approach, I suggest checking out our blog (https://www.tmrow.com/blog) and this talk (https://www.youtube.com/watch?v=PAelZb2ZYwI) which might provide a bit of clarity.
That's a very interesting presentation (the youtube video), I'm glad someone is working in this space.
Somewhat related, the presentation has a lot of math, which illustrates some of the points made in the recent HN discussion of the impact and effects of math instruction in Russia (https://news.ycombinator.com/item?id=22941144). Some comments pointed out how French schools rely on math for entrance exams, and this video a clear demonstration of a software startup having a math background. In fact, the startup probably wouldn't exist or even be conceived without the math and modeling of the data they provide.
The main point of the documentary is that the EROEI of the "green economy" turns out to cost more than using fossil fuels directly. In short, the green economy didn't give common people an alternative, it simply gave business people a new tool to mask their intentions while idealistic politicians genuflect to the lie. The lies trickle down through the career pipe line and people go along with unquestioned assumptions out of fear of losing their jobs.
I personally think the documentary doesn't go far enough about the cultural aspect. I'm speaking specifically of the social norms of conformity, sociopathy and narcissism in the decision making classes. We have fucked future generations for short term gains and the only hope the tech community can come up with is to pretend that we will build rockets to go to Mars. When the kids of the billionaires pushing this horseshit get preventable cancers or tangled up in a class war - maybe then things will change (albeit I doubt it). Until then, it's all bean bags, idiocy, shiny tech toys and fluff. Enjoy it while it last. The magic bullet of nuclear fusion doesn't seem to be likely and the social organization to leverage what we have in a responsible manner doesn't either. We will keep doing what we are doing until nature bitch slaps us and we have to change. Hopefully I'm wrong. I want to be.
It took me a very long time (years) to come to terms with the idea that as a techie I can't do anything to help climate. Tech projects is just an excuse to stay in my comfort zone. A very uncomfortable realization.
Do you not think about living by your values as a matter of integrity?
I don't not steal to stop others from stealing. I don't steal because I don't want to hurt others. If everyone in the world stole, I still wouldn't. If the whole world pollutes, I'm still going to avoid polluting as much as I can.
I would like to add this article by Bret Victor: "What can a technologist do about climate change? (A personal view)" http://worrydream.com/ClimateChange/
This piece by Brett Victor gave me the push to focus more on what work I do rather than just focusing on personal wealth + career. I ended up being employee #1 at solar forecasting company that is targeting the intermittency problem of renewables.
If you’re in a privileged enough situation, I’d highly recommend renewable energy sector, lots of problems where applying technical skill can be extremely impactful.
> Likewise support Green New Deal politicians in the US and elsewhere.
If we are really trying to avert climate change, how is spending trillions on labor unions and guaranteed jobs going to help?
If this was the plot of the movie Armageddon, and there was an asteroid hurtling towards earth, would your planned respond spend pages talking about unions, wealth inequality, indigenous people, and guaranteed healthcare, jobs, and housing? If climate change is really imminent and existential threat that requires “war time mobilization” then how can we afford to water down our response with these tangential things?
Under the Green New Deal you’d spend trillions more a year on healthcare, housing, etc. For that money you could buy up the major oil companies like Exxon (market cap $300 billion) and keep the oil in the ground. You could make the US carbon neutral using existing or near-term CO2 recapture technology. You could bankroll India and China to keep their fossil fuels in the ground and build out fully renewable capacity. You could invest in renewables technology, nuclear, carbon capture at the same time. If you sincerely believed that climate change was an existential crisis, why would you divert 90%+ of the force of the “war time mobilization” on social programs?
In my view, the Green new deal has done incalculable damage to the environmental movement. It is a transparent and cynical attempt to piggy-back traditional left wing politics onto the environmental movement, undermining its credibility and seriousness.
The logic behind the GND is simple: if we don’t do something about this, we’re going to be screwed. People don’t want to do anything about it because they’re afraid it will cost jobs. So make a plan that guarantees them jobs if they do something about it.
It’s a political solution to a technical problem. You may not like the politics, but it’s pretty damn logical.
> The interesting thing about the Green New Deal, is it wasn’t originally a climate thing at all,” Chakrabarti said to Inslee’s climate director, Sam Ricketts, according to a Washington Post reporter who attended the meeting for a profile published Wednesday.
> “Do you guys think of it as a climate thing?” Because we really think of it as a how-do-you-change-the-entire-economy thing,” he added.
The question is whether there is a logic to it beyond the pretext that motivated it originally. And I don’t think there is. Protecting jobs from displacement as a result of climate change mitigation efforts makes sense. That could be accomplished for a fraction of the price of offering guaranteed housing, jobs, healthcare, higher education, etc.
We are talking about spending trillions a year on social welfare to get maybe $100 billion (at most?) on climate change mitigation. Is that “pretty damn logical?”?)
I think it’s one solution on the table, and not even one that has the full support of a single party. The others so far seem to be pretty close to a no-op. Be nice to see some other folks try to modulate it and offer a better proposal.
