Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Introducing PackageManagement in Windows 10 (technet.com)
361 points by sz4kerto on April 29, 2015 | hide | past | favorite | 218 comments


I was going to say, "Finally! They've gotten it right after all these years of sucking so hard as to be unusable on servers." But, I did some digging into the github for OneGet (https://github.com/OneGet/oneget)...

And, they haven't gotten it right. Windows package management still sucks so hard as to make the OS unusable on servers (and annoying as hell on the desktop). This is seemingly a simple-minded downloader, with none of the capabilities I would expect of a modern package manager. Sure, it's nice that it can download and install stuff from the command line (and it's idiotic that it's been so difficult to do that in the past), pulling from a variety of sources. But to imply that this is somehow comparable to apt or yum is disingenuous, at best. I'm guessing it's simply ignorance; Windows folks usually have no idea what they're missing (if they did, they couldn't possibly tolerate how bad it is).

I can find no mention of dependency resolution, for starters, which is the core reason apt and yum are magnificent. Nothing about querying for what files or commands are provided by which packages. Basic validation tools (i.e. if I uninstall or upgrade X, will Y break?), also seem to be missing.

In short: It's simply not a package manager in the sense we mean when we speak of apt or yum. "Lipstick on a pig" is a saying that might apply here.


Is application dependency management in windows normally a problem? Apart from the odd runtime, I expect Windows applications to be pretty much self contained, and not dependent on any system-wide libraries (which is a huge benefit I think).

For development it's a different story, there you can have several levels of dependencies, but there NuGet at least appears at least as powerful as similar options on other platforms.


> Is application dependency management in windows normally a problem?

It has its own name.

http://en.wikipedia.org/wiki/DLL_Hell


DLL hell certainly used to be a big issue in Windows. But I disagree that it remains a major factor; Windows apps are increasingly bundled and Windows system functions that used to be the cause of these issues, like DirectX, are packaged as redistributables that allow side-by-side installs.


How do they handle security updates? Does every app have to update all its own libraries in an e.g. heartbleed scenario?


There's the WinSxS and the GAC (Global Assembly Cache). In SxS actually means side-by-side. Both WinSxS and the GAC supports multiple versions of the same libraries, and the program manifests set up policies for which library versions they require/accept.

An application may for instance say that it accepts any newer "minor" version.

For security patching you can replace a library with a new "build". So if an application uses libraries placed in WinSxS or the GAC it will pick up any centrally patched libraries without the risk of jumping to a too new version with breaking changes. The new version can exist alongside older versions and they can all be patched - for security reasons or otherwise.

A vendor can also distribute libraries alongside the application - in the application library itself. In that case the responsibility for patching is placed squarely on the vendor of that application.


Not sure who "they" is here, ut in windows: pretty much, yes. You can't have it both ways. So you either have a centralized/shared lib environment where you may run into conflicts but you get centralized patching, or you have self contained applications where you need to patch each one but you never have lib dependency/upgrade issues. Simple as that.


Exactly, thank you.


> like DirectX, are packaged as redistributables that allow side-by-side installs.

No wonder everything gets so messy in the registry :)


"but there NuGet at least appears at least as powerful as similar options on other platforms"

That seems unlikely (i.e. can we upgrade the whole OS with NuGet?), but it does look like it is, at least, aware of some of the problems an actual package manager solves.


Not sure if I was unclear, I was referring to NuGet package management for development, comparing it to e.g. similar systems such as npm.


No, I understood. It seems like NuGet is an acceptable substitute for npm...but neither is an acceptable substitute for apt or yum (whether for development or whatever).


Just want to note that Windows has a whole mess of dependency-management, much of it provided by the service that handles those MSI installer files.


Haters gonna hate. Where comes all your frustrations from? I've never had any issues with dependencies on Windows. If some C++ redestributables are missing, they are commonly installed by the installer. .NET Framework the same.

For development NuGet.


And when a library has a security update, the software which contains it does not get updated, because it's repackaged by 10,000 different software companies that don't care or don't have the resources, and they sit side by side, and disk usage is way bigger than linux. This is partly fixed by the fact that installing software is such a pita and hard to automate that you don't install much 3rd party software on windows.


package management is going to suck on windows as long as permissions sucks on windows.

multiple times, i've been unable to delete my own damn files as administrator on windows machines, leaving me angry and frustrated. i've also been stuck in a catch-22 in the extended acl/permissions dialogs, unable to exit. all the googling in the world couldn't help me - it was just fuckin' broke.

sane package management can't exist on such a system. an assertion impossible to prove, but i'm right so far.


As I understand it, one of the difficult problems on Windows is the file locking semantics. When a library is loaded in memory it cannot be replaced on disk. So updates require a reboot to put new files into place. This might be the cause of the effect you describe (administrator being unable to delete files).

It has been about ten years since I maintained some Windows installation packages for a reasonably large project (originally built with a proprietary package builder when I took it over from the previous maintainer and then I ported it to build using a fork of the Python MSI builder). So my understanding of Windows package management and the pains of making things work reasonably is weak and out of date. So take this guess with a grain of salt. I might even be describing what happens and why wrong I dunno...I avoid working on Windows as much as possible, so my ignorance is high.

I can really only speak to the comparison of OneGet to apt or yum, which is absurd on its face. And, I have observed the interesting level of denial that kicks in whenever criticism is leveled at an OS based on weak package management. There's always a lot of "well, I just do this ridiculously unwieldy thing and pretend like it isn't a ridiculous and unwieldy thing to do, so I don't see why I would want something like apt or yum". This tread has been no exception. It's like I'm insulting them personally when I say the workflow they are having inflicted on them by their OS is suboptimal.


As long as there is no single prevalent native package format for Windows, packaging is going to suck on Windows. That's why this tool exists, but I don't think attacking a big mess with more complexity is the right approach. I hope I'm wrong, but I think this will end up 'successful' only in the way NuGet is 'successful' - there's not really any competition.

What I'd hoped for is that MS would commit to extending MSI to handle new use cases and provide actual good tools to make authoring MSI less painful. Maybe some kind of marketing or outreach to get ISVs to use MSI instead of the various alternatives. But for some reason they don't do that, and now it looks like one of the things cut to make Nano Server is Windows Installer (MSI), so I don't think it's going to happen.


Wait. Are you saying that MSFT is going to walk away from the everything-and-the-kitchen-sink installer format that is MSI? This is actually (seriously!) a bit surprising to me!


http://blogs.technet.com/b/windowsserver/archive/2015/04/08/...

"To achieve these benefits, we removed the GUI stack, 32 bit support (WOW64), MSI and a number of default Server Core components."


You probably have far more expertise in this area, but I don't see this as solid evidence that MSFT is walking away from MSI. I mean, they also removed WOW64 and the GUI.

It would be absurd to think that MSFT would ever drop support for the GUI in every version of Windows Server. :)


I don't, my earlier thoughts should indicate that I have less (or none). Thinking about this more it's more likely that Nano Server won't have packages at all: the container will be the package. You'll build the image and it will have whatever it has. If you need to add/remove 'packages' you will build a new image. So, no need for package management inside the container -> no need for MSI.


That's actually a really good thought about the reason for MSI removal in Nano Server!

(And, TBH, my only experience with MSI was briefly working with a vaguely recent version of InstallShield several years ago. The fact that you can talk about the pain of MSI authoring probably indicates that you understand more about the topic than I do. I just clicked some buttons in the InstallShield UI, and skimmed some of the MSI developer docs. :) )


So too little too late?


Perhaps I missed it, but the article doesn't mention this in any way: Windows ABSOLUTELY needs a way to go from 'Fresh Install' to 'Fully upgraded' with minimal interaction.

Linux can do this: run `apt-get dist-upgrade` in Ubuntu, and you get all of the most recent updates.

However, in Windows, you install 3 updates, restart, install 5 updates, restart, install 110 updates, restart....


One of the core issues with breaking this pattern is that you're installing a new feature on this reboot, then installing all of its patches on the next reboot.

  So you install 186 updates on a fresh install. This includes .NET 4.5 (or 4.5.1 or 4.5.2), and requires a reboot.

  Now you're done right? You just installed 168 updates, how many updates can there be? On your next reboot, you get another 60 updates, of which 20 are .NET 4.5 updates and security updates. Another is the OS SP1, which needed a reboot before install.

  Now you're done right? You just installed 228 updates, how many updates can there be? On your next reboot, you have another 168 updates, most of which are patches for OS SP1. 

  Now you're done right? You just installed 396 updates, how many updates can there be? On your next reboot, you have another 12 updates, including...
I've seen this chain across more than 6 reboots. I once literally spent an entire day just patching a server.


This is also compounded by how long those updates take because they somehow managed to make writing files to disk a CPU-bound process at some point in the mid-2000s. If you have 50MB of updates to install on any other operating system it will take a few seconds but on Windows XP+ it will run at a few hundred KB/s while using 100% of a CPU which can run SHA-512 at well over 1Gbps.


Yep, this is exactly what I'm talking about.

I don't see why they cant 'look ahead' to say; Oh- this update X depends on update Y, but it's going to be installed, so we could just add it in the queue anyway.


Or rather, slipstream future updates to current downloads. E.g., if you're installing component X that has patches A, B, C, then perhaps you should get customers to download component X that is already in a patched state.


What I always wonder, while twiddling my thumbs during the interminable upgrade-reboot-upgrade cycle, is how many of these these updates will be overwritten by a later update. Am I wasting my time downloading multiple versions of the same file?


These upgrade-shortcuts could introduce a significant increase in testing complexity if you want ensure that you can go from any system version to the current version.

On the other hand, with the current system people upgrade from Windows 1.0 to 8 (e.g. https://www.youtube.com/watch?v=8WP7AkJo3OE).


Unfortunately the update reboot cycle is incredibly reliable, which is why it has persisted. Can't beat "millions of machines have taken these steps".

Thoroughly agree that a more simplified method would be awesome though.


That's the kind of functionality I expect from Windows Deployment Services.


You can skip maybe half of this by installing the Service Pack manually right away. It still sucks, but not quite as much.

No idea why Windows insists on installing individual updates before the Service Pack that includes them all anyways...


that was decent work. Good job - really accurate.


I'm going thru the process as we speak. :)


I use the powershell script from the packer winrm project.

it's ugly as sin and requires many many reboots (and is slow as fuck) but at least it gets them all installed.


An addendum to this is that Windows needs to restart way too often compared to Linux. When I apply my daily yum updates for CentOS, only kernel updates, which are very infrequent, require a reboot.

Every time I do Windows Updates, I need to reboot. This is highly annoying and IMO makes Windows Server YYYY completely unsuitable for servers.


Windows Server needs to restart at MOST once a month ("patch tuesday"). If I look at a 2012 Core server, it hasn't even restarted nearly that often in the past year. In fact 9 times in 12 months.

Plus I can schedule this restart at any hour, on any day, I wish. And do. For example I restart mine on Thursday morning at 1 am.

If that really makes Windows Server "unsuitable for servers" then your downtime requirements must be so high that any single server wouldn't be able to meet them (i.e. even with Linux you'd need two or more servers to provide that level of uptime).


Once a month is way too often. Even if your downtime requirements are "so high", once a month is ridiculous.

For the sake of this example, let's assume you have customers that depend on your server being up, for the product/service they're paying you for.

Tell the customers that you plan on rebooting once a month, and see how they feel about it.


In that scenario I'd never have just a single server, be it Linux, Windows, BSD, or anything else.

Being in a situation where you're completely unable to restart a machine without losing money, is being in a situation where you're living on a knife edge regardless. A single hardware failure and you're in deep trouble.

I've been in exactly that situation (the entire company rested on a single Ubuntu server), it was really unpleasant and management didn't do a thing about it until we started getting sector errors which the RAID controller's error handling was struggling to cope with (after that incident and the downtime, management shelled out for four physical boxes, a SAN, VMWare licenses, and a setup with 15 minute rollbacks and dynamic migrations if a physical server died).


his point remains. being forced to run 2 servers so your server doesn't go down once/month is a bit ridiculous.

No, it isn't untenable, yes you can work around it, but that's a problem that shouldn't need to be worked around.


Agreed. Up to a certain point, a single server is going to be considerably more cost effective than multiple machines, as well as requiring less devOps time. (Especially nowadays when you can easily lease a server with, say, 16 cores and 48GB of RAM.) I can recall having to restart our production server exactly once in the past 12 months for a kernel update. Every month would be an irritation for sure.


Is it normal to update server software like PostgreSQL or Redis on Linux without stopping the running process during the update?

Maybe Linux doesn't need to restart the OS as often, but you're still going to have down time, right?


You can update the software without pulling it down. In order to use the update you need to restart, but there's a large difference between restarting a service and restarting an OS. It isn't really analogous.


It takes 10 seconds to restart Windows these days, so it's almost analagous.


Do you have any benchmarks to support that claim? I have Windows 8 and 10 systems which take minutes to reboot on an SSD which beg to differ.


You can see it when you do a video search for: Windows fast boot. Here are a couple:

https://www.youtube.com/watch?v=p_vUZR2RtRs

https://www.youtube.com/watch?v=xkKoggifsE4

(And you don't need to log in for your background services to start.)


Neither of those support your original claim. The author is focused on one specific feature, which is nice but excludes the entire shutdown process and booting up until the Windows logo appears. Anyone who's run a server knows that just the BIOS initialization is often much longer than 10 seconds. And, of course, many services take a noticeable amount of time to startup so time-to-available is longer than time-to-login-screen.

Worse, the conversation was actually about the time needed to install updates. Now that Windows logs out before installing updates, you have a fairly sizable time delay – tens of minutes after a large update even on an SSD – while the service is unavailable but before the system starts rebooting.

How does Fast Boot work? Well, it's actually not booting at all – it's logging out all active users and hibernating:

http://blogs.msdn.com/b/b8/archive/2011/09/08/delivering-fas...

That's a really nice bit of work in normal usage but it means that you have to do the standard cold boot process when you're getting a system update. That means that a sysadmin would need to review each update to know whether it would trigger a fast or slow boot, assuming that a patch for e.g. SQL Server wouldn't always trigger a cold boot to be conservative.

The alternative is what everyone's been saying, namely that you have to assume that a reboot is slow and have n > 1 servers if uptime matters.


I understand all of that and I agree with you.

However, if people are going to make irrelevant points about "not having downtime with Linux" then I am going to fuck with them.


That is BS. When you install updates it restarts to install them. The time is proportional to the number of updates.


A server is just a name. Just because a host machine needs to go down doesn't mean that the "server" has to go down.


Popular (gaming) services have weekly downtimes to patch servers, and while users are annoyed, they seem to get over it and keep playing just fine.


It's completely insane to have multi-hour downtime every week the way Blizzard does. Between that and their regionally segregated Battle.net (there's no good excuse that you can't play Hearthstone with anyone anywhere), they're basically stuck on server architectures from the 90s.


Windows Server needs to restart at MOST once a month ("patch tuesday").

BUT if Windows patches came out as often as CentOS patches do, I'd been rebooting every other day.


This is the approach taken by LY-Linux ("patch leap year"), although better implemented IMO. In fact, I typically only have to restart my servers once every 4 years (at most).

Look it up, it's awesome. (Please don't.)


This might be the case if you run servers on single machines rather than in clusters. And if you are running servers on single instances, aren't you already giving yourself a SPOF? If you need a specific server to be running at all times it seems like you have a problem.


What I really don't understand is why Windows Update takes so long to apply each update. I wouldn't mind 200 updates and a couple of reboots if it only took 10 minutes to go through them.

In most Linux distributions, an update simply unzips a tarball somewhere and runs some bash scripts. Unless it's a very large package, each update usually only takes a fraction of a second on SSD, and maybe few seconds on HDD.

Windows, on the other hand, often takes minutes installing each update. Going from freshly-installed to fully-updated can easily take several hours, even on an i7 with SSD and a very fast connection (so the download speed is not a factor). It's even worse on an older computer; I've seen a smallish .NET update taking over 10 minutes to finish, and there were several such updates. What the hell is it doing behind the scenes that takes so much time?


>restart

This isn't going to happen. MS is tied deeply into restarting due to how it loads libraries and other issues. Unless they write a new OS from scratch and give up on legacy compatibility, well, don't hold your breath.


I don't even mind the multiple restarts, the part that bothers me is that I have to babysit Windows Update as soon as my machine finishes restarting, tell it to get updates, tell it to install now, and then have it restart again. If Windows Update could collect all the update packages, and I could leave it alone for 20 minutes and come back to a fully updated system, that would be fantastic.


Yeah, this seems like it should be pretty straightforward (I'm sure it is harder than it seems, but still). Set a flag that persists through multiple reboots that says "after Windows loads, if there are updates, apply them and reboot, repeat, otherwise unset the flag".


Fun fact: I helped tweak up a Powershell script that does this exact thing: https://github.com/tomswartz07/packer-windows/blob/master/sc...


I owe you a reasonable beverage of your choosing. Please redeem anytime you're in melbourne.au :)


If only it had a place to register a setting or store a file containing information like "update until you reach the latest version". It'd also need to be able to get through all the updates without crashing even once (otherwise we'd need to check the disks and repair start-up files, and that can't be automated), so maybe this is just impossible.


Even in Linux you've got to restart once, if you update the kernel or somesuch. The problem with Windows is the restarts before the final one, where you need to finish upgrading to version N+1 before the updates taking you to N+2 will even download.

If the updater software could just plan out a patch-graph ("once I do X, then Y will be available, so I'll do Y, and that means Z will be available"), then apply all the updates it had recursively discovered and only restart at the very end, everyone would be happy.

Even though the resulting mechanism wouldn't be very similar at all to Linux's idempotent "just install the latest version of the package, you don't have to take the steps in-between" system, it's the UX that matters.


Actually the 4.0 release has added support for no-reboot kernel patching, IIRC.


Heck, I'd be okay with booting into a Windows PE drive, if it meant that after 1 reboot I'd have a fully upgraded OS.


If this day has taught anything: MS is moving to Linux


A lot of the time, restarts are required to remove old versions of files that are still in use. AFAIK, there is no way to remove files that are 'in-use' (and not locked) without having to make an entry in the registry to schedule a deletion on the next reboot.


I always wondered about that.

Shouldn't a real OS and filesystem be capable of removing something in use?

IRIX always could. So can Linux a whole lot of the time, if not all the time, as can BSD and Mac OS.

Here's something:

Once I upgraded a pile of IRIX boxes to a new version of some CAD software. A few of them had the older version running...

Didn't matter!

The files got cached, and those versions continued running the next day, until shut down. Next launch, the new version popped up.

One could just run a program, then delete the program right out from under the running process, and that process didn't know. IRIX just cached it all, and when the process was done, cleaned up the cache.


> Shouldn't a real OS and filesystem be capable of removing something in use?

It's a choice. Windows specifically uses DLLs as non-dirty swap, i.e. if the memory manager want's to free up pages it may free memory pages used for mapping a least recently used DLL because it will not require it to write any "dirty" memory. A DLL page can never be dirty. Windows knows that it really easy to read in the image again (from the DLL) - so it can just kill the page and reuse it immediately. But then it has to know that it can get the exact same memory page back from the DLL. If the DLL has been deleted or changed that would lead to some strange errors.


This is no different from the situation in Linux, BSD etc, where executables and libraries are simply mapped into the process with mmap() - they're backed by the files and the system is free to discard those clean memory pages as it sees fit.

When you install(1) a new version of a file, the old one is deleted and the new one written in its place. The file handle in existing processes continues to point to the original, and the filesystem will wait until everything's closed it before actually discarding the file data.


Thanks for clarifying. That boils down to linux perfectly hiding this problem from the user, while windows painfully makes the user restart every time a file is in-use and needs to be removed.


Thanks.


Solving this is one of Matt Wrock's goals with Boxstarter. I've been experimenting with it recently and found it quite nice.


The way around this is to update your deployment images from time to time.

It's one reason I liked service packs, new service pack comes out, generate a new install image.


Right, and that takes care of the 'big ones', but you still have tons of other minor updates that are released every Patch Tuesday.

I currently have Jenkins build an OS image every two weeks, just so our Deployment Server has an up to date image.


> I currently have Jenkins build an OS image every two weeks, just so our Deployment Server has an up to date image.

Ya, lots of tools have made this easier recently, but thinking about it, doesn't WSUS handle this?

I haven't been a Windows admin for a while, but I thought with WSUS it was install, join and forget while it did it's thing. Maybe it's been too long ...


WSUS just caches updates locally, so you can download once and have the WSUS server distribute copies to your forest.

It's a GPO that points your domains to the WSUS server. And I've only used Octopus to automate WU installs.


> WSUS just caches updates locally

New updates do supersede others though, if they're marked to do so from Microsoft. That should cover at least some of the .Net point release updates and the like.


> released every Patch Tuesday.

It is the second Tuesday of the month, not every Tuesday. They've only broken that pattern a handful of times in the years that they have been doing it when an "emergency" patch was required.


>> released every Patch Tuesday.

>It is the second Tuesday of the month

that's exactly what "every patch tuesday" means (notice the "patch" adjective).


I'm guessing this is the plan with the new system of "builds" we've seen in the Technical Preview. Just install the month's build and then the few updates it needs after that. Fingers crossed, anyway.


Yeah I think they need to address this but it's going to require a pretty big, fundamental change so it might take a while. It's certainly needed.


Remember the Windows 95 days?

    Your mouse pointer has moved. 
    Please restart Windows for this change to take effect.


I got "You must restart your computer to apply these changes" in Windows 8.1 just by moving my mouse to another USB port a couple of weeks ago. I guess some other non-critical update was queued up.


That's exactly what they're trying to do with the new app model.


I think he's referring to system updates, not just installed software.

My usual workflow for setting up a Windows machine goes like this:

  1. Install Windows
  2. Download updates
  3. Reboot
  4. Download updates
  5. Reboot
  ...
  28. Download updates
  29. Reboot
  30. Download updates
  31. Reboot
  32. Install software
It's not great. OS X has been doing cumulative updates from 10.x.0 to 10.x.current for what, a decade now?

MS is rebuilding a lot of essential utilities in their "modern" app framework (including the start menu), but as far as I know, the lower level system updates still get distributed the same way as before, and frequently have long chains of prerequisites.


Although I applaud the efforts, at the current state that's just sugarcoating the nastiness of code-executing installers. A package installation should basically just deploy the components of a passive package archive to the filesystem and leave any actions to the package manager. This solution doesn't even provide a generic interface e.g. to run a query which file would be provided by which package or to which package a specific file in the filesystem belongs to. The installation process still relies on mostly uncontrollable code execution.


Well, how else would windows software give you an option to decline to not skip disabling the installation of their system heath turbo-boost quick launch tray toolbar search engine registry cleaner by checking a box?


Windows badly needs this to be a general Development OS. Lot's of developers in the last years run away from Windows to OSX and Linux because of this.

Without Visual Studio, which installs the C++ and .Net dev env, it's a pain to install anything which works out of the box in Windows.

Example: Perl or Python both have an installer but you have to update the system PATH, the console sucks so you can never use it, no simple way to keep it update unless you use the installer again and so on...

If all these goodies are available in a "unixy away" i'm sure Windows will be used by general Developers.


>> "Without Visual Studio, which installs the C++ and .Net dev env, it's a pain to install anything which works out of the box in Windows."

On Mac don't you need to installed the Xcode command line tools before being able to do most dev stuff?


Which can be done by running 'clang' from a terminal. And if you want the full Xcode, install from the App Store with just an Apple ID needed, again for free.


That's the easiest way, but not the only way. Likewise, on Ubuntu you also often need build-essentials and various other dev libs manually installed.


Do we really want to compare "sudo apt-get install build-essentials" with installing all VS, Perl,Python etc... in Windows ? :)


Well, in some ways it's worse.

I'm not too familiar with linux, but a year or two ago, I was trying to set up a rails development environment on a Ubuntu VM. I had quite a few challenges along the way, but one of them that I recall that confounded me for an hour or so was that I needed to install build-essentials, which was not documented in the guide I was using, nor was it marked as a dependency in whatever package I was actually trying to install.

When you install Visual Studio, generally it automatically installs anything you're missing, or at least tells you how to do it.


I am a Java dev and don't use VS and things basically work "out of the box" for me. I used to be a big "Linux is the best dev env" kinda person but I've changed my mind since I started doing Window's development - it really is just as easy - for the stuff I do.


> Without Visual Studio, which installs the C++ and .Net dev env, it's a pain to install anything which works out of the box in Windows.

Depends on what you're working with. I don't have VS on my Windows machine at home, but everything tends to work out of the box. Granted, I see plenty of installers with the VS Runtime redistributable, but that's not proper VS.


I recently blogged about the Linux world needing something like this.

http://leto.electropoiesis.org/propaganda/plugins-and-packag...

I hate having to individually update my Wordpress install, my Rubies, my system packages, my IDE (be it Eclipse or Android Studio), a separate TeXLive install from the OS packages, even Vim now has its own package management with Pathogen...

There must be some way to _unify_ this proliferation of software update mechanisms.

edit: It would be ironic in the extreme if Windows of all platforms manages to get this out the gate before the Linux community considering how awesome stuff like apt-get, yum, and emerge are... Oh well, just goes to show that open-source giveth with the one hand and taketh with the other.



So cynical. But unfortunately, so true.


Ha! That's exactly the first thought I had. I guessed someone would beat me to it :-)


>There must be some way to _unify_ this proliferation of software update mechanisms.

https://gnu.org/s/guix

https://nixos.org


Unify != replace. Guix/Nix don't integrate multiple package managers, it just is Another Package Manager.


Nix runs on any linux distro because all packages are stored in /nix, so it's not trampling all over your files in /usr, /bin, etc. It also runs on OS X (packages work, but are a bit less well tested, and the binary replacements are currently not as easily available as they should be). In the past it has been made to run on BSD and Windows through Cygwin, but these need a lot more work before being production ready.

It's also significantly more flexible than other package managers due to how it can handle multiple versions of the same package in an elegant way. This makes unifying diverse package sets an actual possibility. Apt, yum, etc are too limited to unify package sets in this way, an creating another layer above them in order to integrate them is hacky, ugly, and unlikely to work well.


They are the only package managers I know of capable of unifying package management with a single set of tools. Unifying multiple distinct package managers is a fool's errand.


I don't see how they can. Guix/Nix don't run on Windows, so they can't replace the NPM or PIP or Ruby Gems, first off. But I don't even see anything about Guix/Nix that make them especially capable of unifying Linux package management. (Heck, given that there's two of them, I don't see how either can be said to unify the specific kind of Linux package management they do.)


>Guix/Nix don't run on Windows

Correct. On Windows you can't do any better than a swarm of language package managers. Without the ability to create a custom Windows distribution starting from a systems level package manager, there's no hope in having anything better.


> Unifying multiple distinct package managers is a fool's errand.

I'm really hoping that isn't the case.


> Unifying multiple distinct package managers is a fool's errand

I disagree. Why do you think that is the case?


Different package managers make different design decisions that make them incompatible with each other without sacrifice. Guix/Nix focus heavily on reproducibility and not relying on any third party binaries. These features would have to be thrown away if it unified pip, npm, etc. because they make no such guarantees.

Every package manager works different, and trying to accomadate all of them with a unifying tool will require a lot of time wasted writing interfaces between them all. I don't see a way to do it without lowering the feature set to the least common denominator and settling with that. The real solution is to elevate our system package managers to the point of handling the important use cases that currently only language package managers provide, such as virtualenv/Bundler style management (i.e. installing packages somewhere besides /). Nix/Guix accomodate all such use cases, which is why I promote them.


> These features would have to be thrown away if it unified pip, npm, etc. because they make no such guarantees.

Again I disagree. For example in this case, it would just mean that when installing from nix repos you get the reproducibility etc guarantees, installing from upstream repos would get you vanilla version and installing from OS repos would get you an integrated/patched version. But the installation process (from user point of view) could still be unified, because regardless what happens behind the scenes the high-level stuff is pretty much the same; nix -i/apt-get install/pip install could definitely be unified under one umbrella tool.

> I don't see a way to do it without lowering the feature set to the least common denominator and settling with that.

And the common denominator would probably cover 80-90% of uses. For the rest you'd still have to option to delve deeper and use some implementation-specific tools if necessary.


I just think it's a mistake to believe that the solution to the proliferation of package managers is to write a package manager manager.


I'm not sure that anybody is advocating taking that route as the way forward...


If we've any hope to unify package management, we need to get to the essence of what package management is. It's really quite simple though - it's the ability to say that one piece of software depends upon another, and to have a piece of software which can automatically resolve the dependencies (which form a DAG). To construct our DAG we need a list of nodes (the packages), and a list of edges (the dependencies of a package).

If we say that packages are basically just binary blobs of data (say, a .tar.*), then we might construct our database to identify (key) our package payloads. I'll use some pseudo pgsql for illustration purposes.

    CREATE TABLE packages
    (
        payload bytea NOT NULL,
        package_name character varying NOT NULL,
        CONSTRAINT pk_package PRIMARY KEY ("package_name")
    );

    CREATE TABLE package_dependency
    (
        dependant character varying NOT NULL,
        dependency character varying NOT NULL,

        CONSTRAINT package_dependency_dependant_dependency_key UNIQUE (dependant, dependency)

        CONSTRAINT package_dependency_dependant_fkey FOREIGN KEY (dependant)
            REFERENCES packages (package_name),
        CONSTRAINT package_dependency_dependency_fkey FOREIGN KEY (dependency)
            REFERENCES packages (package_name),

    );

Simple. But we're mising a bit here. We need to update software, so a name is not sufficient to identify a dependency. Lets add that.

    CREATE TABLE packages
    (
        payload bytea NOT NULL,
        package_name character varying NOT NULL,
        version integer NOT NULL,
        CONSTRAINT pk_package PRIMARY KEY (package_name, version)
    );

    CREATE TABLE package_dependency
    (
        dependant character varying NOT NULL,
        dependency character varying NOT NULL,
        dependant_version integer NOT NULL,
        dependency_version integer NOT NULL,

        CONSTRAINT package_dependency_pkey UNIQUE (dependant, dependency, dependant_version, dependency_version),

        CONSTRAINT package_dependency_dependant_fkey FOREIGN KEY (dependant, dependant_version)
            REFERENCES packages (package_name, version),
        CONSTRAINT package_dependency_dependency_fkey FOREIGN KEY (dependency, dependency_version)
            REFERENCES packages (package_name, version)
    );

Cool, now we have a composite key we've got sufficient information to idenfity a dependency right? Well no, we now have the problem that the same piece of software with the same version could be distributed by different vendors (with different dependency chains/configurations, etc). We had to modify the original solution to get here rather than extend it. Let's modify it again!

    CREATE TABLE packages
    (
        payload bytea NOT NULL,
        package_name character varying NOT NULL,
        version integer NOT NULL,
        vendor character varying NOT NULL,
        CONSTRAINT pk_package PRIMARY KEY (package_name, version, vendor)
    );

    ...
Great, now given a combo of package_name, version and vendor, we can uniquely identify a dependency without worry. All problems solved?

What now if Vendor has a customer with different needs, and must distribute two different derivations of the same package version and number? Do we add another field for "configuration", and if so, what type do we make it? Do we just rename the package and lose the relationship that exists between them? It should be blindingly obvious by now that we're just trying to add structure where it isn't really present, and we're making the solution to the problem more and more complicated.

Now take into account the possibility that Package Manager A implements dependencies using a tuple of (package_name, version, configuration), and Package Manager B implements dependencies using a tuple of (package_name, version, vendor), then if we want to unify these models under a "one true package manager", then our OTPM needs to model things using ("package_name, version, optional[vendor], optional[configuration]), and so forth. Multiply by N package managers with their own individual quirks and you get a "unified" model which is barely unified at all, the only structure to it is really our initial solution - packages with names.

Here's a reduction in complexity: Instead of using a name, which may be ambiguous and therfore requires us to constantly add fields and change the underlying model - let's propose we have some means of creating identifiers with some means of guaranteeing uniqueness. We can basically go back to our initial model:

    CREATE TABLE packages
    (
        payload bytea NOT NULL,
        identity uuid NOT NULL,
        CONSTRAINT pk_packages PRIMARY KEY (unique_id)
    );

    CREATE TABLE package_dependency
    (
        dependant_id uuid NOT NULL,
        dependency_id uuid NOT NULL,

        CONSTRAINT package_dependency_dependant_id_dependency_id_key UNIQUE (dependant_id, dependency_id)
 
        CONSTRAINT package_dependency_dependant_id_fkey FOREIGN KEY (dependant_id)
            REFERENCES packages (identity),
        CONSTRAINT package_dependency_dependency_id_fkey FOREIGN KEY (dependency_id)
            REFERENCES packages (identity),
    );
Now, if we want to integrate "Package Manager B" into this new model, we can extend our system. (Note keyword extend, not modify). We can implement a new table which references the base model.

    CREATE TABLE pmBpackages
    (
        identity uuid NOT NULL,
        package_name character varying NOT NULL,
        version integer NOT_NULL,
        vendor character varying NOT NULL,

        CONSTRAINT pmBpackages_identity_fkey FOREIGN KEY (identity)
            REFERENCES packages (identity)
    );
And as far as pmB is concerned, this is just an implementation detail - we can hide it from any users and just present the legacy view to them.

    CREATE OR REPLACE VIEW "pm_B_view" AS 
        SELECT package_name, version, vendor, payload
      FROM "pm_B_packages" NATURAL JOIN packages;
So here's a challenge. Begin with the schema for "Package manager A" as implied above, and try to implement "Package manager B" by extension (not modification). The first point of struggle might be to notice that you need to invent "configuration" values, since "Package Manager A" requires them as part of the key, and they're NOT NULL.

Hopefully it becomes obvious now why trying to build another model on top of other overcomplicated models is the real fools errand, because none of them so far have understood the essence of the problem.

I've glossed over how we might guarantee uniqueness for package identities so far. The solution is to use a cryptographic has of the payload, under the (fair) assumption that a modern hashing algorithm is sufficiently collision resistant. There's nothing language/framework/operating system specific about SHA-1 or whatever the choice of algorithm.


I think you are massively overthinking this. Dependency tracking and all that jazz are just implementation details of individual backends, the unifying tool does not need to be aware of that at all.

Let's say we define basic API with two verbs: SEARCH and INSTALL. When the user wants to install a package the unifying tool first queries backends with SEARCH to see if they have such package, and after resolving which source/backend it wants to install it from (a process that might involve user interaction), and then invokes INSTALL to that backend. Nowhere in this process the unifying tool needs to know what black magic the backend needed to do to get the package installed.


SEARCH is the reason package management is so broken. If a developer intends a particular piece of software to be installed as a dependency, he should convey that to the users, rather than a vague criteria for which the user might hopefully get the right thing. The search needs to return only one result - the right one. If several package managers keep a package of the same name and version, we shouldn't need to keep adding criteria to narrow down our search until we get the right one (and someone could later add a package which meets all those criteria after you've published).

The heart of the problem is making a SEARCH which will always return one result, and to do this we either need to add N criteria which are collectively guaranteed to be unique - or to just stick a unique identifier there to begin with and treat the rest as information queryable from the content.

The use of a hash as an identifier solves several other problems such as having multiple repositories whereby each one could host packages of the same name and such. While we continue to rely on search without some means of uniquely identifying packages, we continue to have the social burden of making sure our repositories don't conflict with other peoples (usually the "official distro" repo - but when we're talking of unifying packages, that's a lot of repositories which could conflict. Unless we add our "vendor" flag into the mix, etc)


That's all fine and well but the thing is -

Is it possible to unify package management un-intrusively or not? Call the un-intrusive path A nd the intrusive path B.

A) If we were to to it in such a way that did not require us to modify all the existing package managers out there then how do we do that? I think the solutions (a database of unique hashes and so on) you are coming up with might be a way forward. But don't we still have the problem of communicating with the underlying package managers?

B) What sort of intrusion? Would package managers have to adhere to some kind of standard or API or expose a minimum amount of surface area? How would you get package manager maintainers to sign up to something like that? A summit? Who would fund such a summit? Then we'd still need something to implement that API. Somebody elsewhere mentioned PackageKit, would this fit the bill? If not, is there something else that would? And would we still need to track installs and so on with something like what you're proposing.

Remember this would ideally would on Debian-like (.deb apt-get) and RPM-like (.rpm yum) and Gentoo (.pkg emerge) and so on and so forth ad nauseum ... I mean, I'm running Ubuntu so after the Great Request for the Unification of Managing Packages Summit (GRUMPS) I want to still be using Synaptic but I want to be able drill down into sub-package managers. See what I mean?


Again I think you are overthinking this. It is perfectly fine for SEARCH to return multiple results, it is intended mostly for interactive use anyways. It won't impact dependency resolution in any way, that would continue to work as it does now, in other words each backend does its own dependency stuff without any regard of any other system.


It's a disaster. The proper way to do it is distro packages, but for some reason every language, framework, ecosystem and individual developer wants to reinvent this particular wheel. I really don't understand why.


It's because individual tools usually don't want to be tied to assumptions made by one particular distro. I actively avoid using distro packages for 3rd party development libraries and such, especially when a good tool for accessing upstream sources (eg pip) is available.

I use packages for certain tools and platforms, and libraries if I feel the library is really something I want to be a standard part of the system environment. For example, I am more likely to use the distro package of a python library (if available) if I'm planning to use the library for a system administration task than if I am planning to use it for application development. I'm also likely to use distro packages for things like apache, nginx, postfix, unless I have some case-specific reason not to.


One technical reason is that I might use two different versions of the same library in different projects and apt-get only allows me to have one at the time. I think npm and gem are brilliant on this regard.

Best of both words: docker. I consider docker an application packager.


Docker is definitely one answer.


Distro packages move slower than people would like so they create their own solution.


You know, it is still simpler to make your own deb or rpm, than entirely different package system.

It is more of a case, that these different package systems were introduced on platform that lacks native one. Then, by combination of laziness/not wanting to build another package and recycling the already built binaries, they got a traction on linux systems too.

And there is a reason, why distro packages move slower - people having that deployed in production do not like breaking changes. If you want bleeding edge packages, use bleeding edge repos.


There's no need to unify the installation of those domain-specific package managers - those packages just need to provide proper machine-readable metadata to make it possible to build distribution packages out of them so there's no need to create another 10 package-datasilos outside of the system's package-manager's control.

The very idea of placing dependencies in a local location (or even worse in a global location but not controlled by the system package-manager) is so rotten from the core - it simply deserves to die.

What was once supposed to be a tool for developers to have easy access to dependencies has now crept into the area of operation and deploying uncontrollable stuff (Docker containers, 3rd party package managers, statically linked applications, manually built packages, …) into production systems has become the norm.


> PHP Pear

Actually, Composer is what's usually used these days. Composer is per-project (like npm), though, so there's no point unifying it into a global package manager.


See, I didn't even know that about Pear and Composer. And I totally forgot to mention npm and bower from the javascript world, both of which I use.


Have you ever tried ninite (https://ninite.com/) ? Not perfect at all - but at least it works for some of the freeware utilities / programs I'm using on my windows PC.

Simply run the generated exe to install the program. Run it later to update the program.


Looks cool, but I meant for Linux specifically. Apple and Microsoft have ultimate control over their ecosystems when it comes down to brass tacks.


Give it about 6 months and someone will announce a container-based package manager.


Hum, you could be right. I tried to install Discourse from source a while back on CentOS and gave up. A while later I tried the container method (Docker I believe) and bingo. So much magic going on under the hood though, dunno how I feel about it...


This kind of reminds me of just statically linking everything. We have not done that because it is a waste of resources but containers are all right then. :)

(OK, I'm aware of the differences, but I don't see containers as a salvation for client-side deployment.)


https://developer.ubuntu.com/en/snappy/tutorials/using-snapp...

I was informed about this just this morning.


There must be some way to _unify_ this proliferation of software update mechanisms.

There's PackageKit, though that's mostly used for creating those graphical software center/app store types of applications.



> There must be some way to _unify_ this proliferation of software update mechanisms.

There are many ways. The problem is there are many different environments who all use different methods for different reasons.

To a solo developer with personal control over the entire stack, the Operating System is merely one more tool in the toolbox. That dev can pick any distro he wants and then install and configure anything he wants(as root). To a postdoc researcher in a lab using the University's shared compute cluster, leveraging the OS might not be such an obvious and easy choice. Then of course there is the whole issue that Wordpress, ruby, system software, and IDEs are all developed by very different groups of people.

But back to your question: configuration management tools can help with this problem. They do require some end-user investment at this stage, in no small part due to the reasons I identified above (there's not yet an ideal default that works for everyone).

Generally, configuration management tools encourage you to declare your software, modules, packages, requirements in an abstraction layer and then have the config management tool handle the messy details of whether to use apt or yum or gems. The catch right now is that you will typically have to do handle those decisions (to some extent) yourself.

Ansible:

http://docs.ansible.com/gem_module.html

http://docs.ansible.com/apt_module.html

http://docs.ansible.com/yum_module.html

Puppet:

https://docs.puppetlabs.com/references/latest/type.html#pack...

Chef:

https://docs.chef.io/chef/resources.html#gem-package

https://docs.chef.io/chef/resources.html#apt-package

https://docs.chef.io/chef/resources.html#yum-package


Aha! Lightbulb moment. I thought that Puppet and co. (chef and so on) were about pure configuration across multiple machines. I didn't realise they were about deployment as well. In that case I guess the problem is sort of solved but then I need to start thinking a layer higher than I have been.

It would be nice if the solution to this proliferation still allowed me to think at the level of [synaptic/apt-get/dpkg] on Debian/Ubuntu/... or [yum/rpm] on Redhat/CentOS/Suse/... and so on. Do you think this is unreasonable of me?


They should just all be in PPAs on Ubuntu.


Oh, wow.

Please, can I get a decent command line now?


I use cygwin. I run cygwinsshd on localhost:22 and use Putty for my client.

A lot of windows-specific gui commands won't work through the ssh layer, but will if you run the cygwin bash shell directly. (Maybe there's some way to get them to work over SSH but I have never had reason to try and figure it out).

    $ which nmap.exe
    /cygdrive/c/Program Files/Nmap/nmap.exe
    
    $ nmap.exe -PN -p 80,443 news.ycombinator.com | grep tcp
    80/tcp  open  http
    443/tcp open  https
    
    $ which ipconfig
    /cygdrive/c/Windows/system32/ipconfig
    
    $ where ipconfig
    C:\Windows\System32\ipconfig.exe
    
    $ where which
    C:\cygwin\bin\which.exe
    
    $ which where
    /cygdrive/c/Windows/system32/where
    
    $ ipconfig /all | grep Physical
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
       Physical Address. . . . . . . . . : [redacted]
    
    $ reg query 'HKEY_LOCAL_MACHINE\Software\GOG.com\' /s | head
    
    HKEY_LOCAL_MACHINE\Software\GOG.com
        DefaultPackPath    REG_SZ    F:\
        DesktopIcon    REG_SZ    OFF
    
    HKEY_LOCAL_MACHINE\Software\GOG.com\Games
    
    HKEY_LOCAL_MACHINE\Software\GOG.com\Games\1207666383
        PATH    REG_SZ    F:\Star Wars - X-Wing (1993)
        GUID    REG_SZ    {F014338D-42E5-4DFC-BE77-F8E52BD998D5}


Why Putty over mintty? mintty is a fork of PuTTY's UI for local use, and is the default in cygwin these days.


I can't believe in 2015 PuTTY is still a thing.

Other platforms have a half dozen terminal programs, three of which are usually very good, and Windows has one that looks like it was hacked together on a weekend in 1998.


There is no analogy to putty on a Unix platform, because its not just implementing the console interface, its also the SSH implementation as well, on Unix systems, its pretty easy to write a fully functioning terminal, because your just hooking into the existing pty system, so all you need to do is write (or borrow) a competent vt100 emulation, and make a UI.

PuTTY is great, honestly. It's not pretty, but neither is the windows console nor PowerShell.


It's an ssh client. It is robust, easy to install and it works well.

How much energy do you expect me to waste evaluating and choosing new ones when the one I have works just fine the way it is? I have better things to be doing.


This is the poison in the open source world. If you have something that works "well enough" but never evolves, it holds everyone back. The barrier to entry is fairly high for an application like this, so I'm not saying this is an easy thing to fix, but it's worth fixing.

Linux in particular is filled with "well enough" solutions that should be striving to be better. Although there's misfires when controversial things like systemd happen, there has been steady improvement in other areas. Imagine if Nginx had never happened, if NCSA httpd was "good enough"?

The Windows user base deserves better than PuTTY.


But whats wrong with PuTTY?, is it missing features?, I mean. like it has full SSH support, its easy to use, and has a non-quirky terminal emulator.. I guess it could have more terminal types, but there are closed source options for that, like Reflections.


Mostly it seems like overkill to run a ssh server + client to get a local terminal.


I've been using putty since 2007 for managing remote unix systems. It's nice to have a uniform interface for everything. If I ever get to a point where using putty becomes a headache for some reason (and it hasn't in 8 years) then I'll switch to mintty. Mintty is definitely a better way to run the windows gui stuff from the command line if you care to do that (I just use the start menu to run windows gui apps).

Occasionally I get tripped up on linux terminals due to the additional requirement that I "ctrl-c" to activate a copy, but that is extremely minor since that method of copy/paste is shared by almost every other tool I use except for putty.


Copy/pasting long lines of text from the terminal...so tired of fixing all the broken lines manually.


Have you ever tried MobaXterm? I use that as my daily driver shell and am very happy with it! Both for ssh to remote boxes and as a local command prompt

http://mobaxterm.mobatek.net/


I get rendering problems with my font (BDF UM+ Outline), and find it has broken mouse reporting. e.g. in tmux, vertical splits aren't rendered, and clicks result in spam being written to the shell.

Also couldn't feel more clunky if it tried, impressive as its feature set is.


You mean like PowerShell?


No, the one with history going back more than an hour, and tab completions on anything except files in `pwd`.


Try https://github.com/lzybkr/PSReadLine - included in Windows 10 now.


Hey, do you have an announcement somewhere about it being included in 10? Congratulations, at any rate.


It hasn't been announced anywhere unless you count random tweets. I should post something on the PowerShell team blog one of these days.


> No, the one with history going back more than an hour

When did you last try PowerShell?

In version 1.0 the history size was 64 by default - but could be changed.

In PowerShell v3 that's increased to 4096 by default. If you can overflow that in an hour you are a seriously impressive typist! That more than one command line per second - sustained.

If you manage to break that, you can assign any value up to 32767 - yes that's 9 different commands per second.

PowerShell can discard duplicates, so if you manage to break 4096 different commands in the run of one day you are doing something wrong.

Like not using snippets (http://blogs.msdn.com/b/powershell/archive/2012/06/27/snippe...).

Oh - i forget - bash don't do snippets.

> and tab completions on anything except files in `pwd`.

Seriously, have you tried PowerShell - like ever?

PowerShell's tab completion blows anything bash out of the water: o PowerShell tab completions are based on command metadata - not on external "completion definitions". The result is that tab completion works for all cmdlets - not just for those that has a "completion file". o PowerShell will complete on anything: Command names, parameter names (options), parameter values - even dynamic values. Try this on PowerShell "ps <tab>" - and you'll be allowed to complete based on the names of running processes. o PowerShell tab completion also works for parameters of user defined functions - and even for parameters of script files. If you declare a script to take parameters, PowerShell will pick it up and allow tab completion on the parameter names and allowed values.


> When did you last try PowerShell?

I fired it up just now on Windows 7 that I have available.

> In PowerShell v3 that's increased to 4096 by default.

Once I quit Powershell, all my history's gone. On my unix machine, I can search my command history all the way back to 2011.

This short-term memory won't cut it for my needs.

> Try this on PowerShell "ps <tab>"

I tried, and immediately received the first file in my user directory.

"dir /<tab>" will erase the / and complete it with some other file name. I was expecting it to list the options.


that's basically what conemu brings


Disclaimer

ConEmu is not a shell, so it does not provide "shell features" like tab-completion, command history and others. [1]

[1]https://code.google.com/p/conemu-maximus5/


Fair point, my mistake: my ConEmu does have history and copletion but it has been so long ago I installed it I didn't remember this is through clink

https://code.google.com/p/conemu-maximus5/wiki/ConEmu_Clink


Powershell has had that support for quite a while too.


Have you given Cmder a go?

> http://gooseberrycreative.com/cmder/


I think one of Windows 10's improvements will be a better command line.


you can get conemu


Tried it, and it looks nice ... however crashes very often for me, and sometime becomes horribly slow (typing and seeing it displayed takes close to a second..)


I really like cmder (http://gooseberrycreative.com/cmder/) which is a wrapper to conemu + clink (which adds a bunch of nice enhancements to the command line).


Or Console2/ConsoleZ. I've been using them + cygwin for 7 years no, they're quite solid.


This really is an audacious proposal and will be an awesome tool if they pull it off. For now there is still a long way to go. There is no command for upgrading a package version yet for example; in fact, when I looked at this a month ago, if you force it to install a newer package version without uninstalling the old one first, it simply seems to forget the old one ever existed and leaves associated files behind. Hopefully a lot of this will be ironed out by the time Windows 10 is released.


As someone who isn't very familiar with package managers, what does this mean for Chocolatey?


https://groups.google.com/forum/#!topic/chocolatey/a8WdEoF-M...

> What does this mean for Chocolatey? More awesome. More security. More better. Chocolatey is going nowhere, OneGet is a core API that enables package managers to hook in and take advantage of Windows components easier.

> The best understanding I have is that OneGet is shipping with a prototype plugin compatible with Chocolatey. Right now this means that it is separate from Chocolatey. We will be working with Garrett to bring the plugin over to core Chocolatey as we rewrite the Chocolatey client in C# this year.

> Right now folks are getting up in arms as they see this as a replacement for Chocolatey and it is not. Chocolatey will need to build a few hooks for use with OneGet.


So Chocolatey is gonna use OneGet in the back? I was thinking it would be a package provider for OneGet instead.


I don't know much about how this works, but it looks like that's exactly it.

https://github.com/OneGet/ChocolateyProvider


Well, I don't feel like it really had any future. Sysadmins knew MS was going to launch this thing soon and Chocolatey requires trust in some random third party, which is unsettling. My understanding is they recently tried a more linuxy volunteer based system, but frankly its too little too late. I'd say once this launches Chocolatey is dead. I guess it could work with this system, but why bother with all of its liabilities? You can just build a local package source on a shared drive and call it a day. Or I could see an official one eventually hosted by MS. Enterprise needs some level of protection against poisoned sources.

Also, word to to wise, if you want to push your software in an environment full of regulations, PHB's, etc pick a professional name for it. There's a reason people pay for Photoshop or use Paint.net instead of The Gimp. Or why the FCK editor goes by the name CK editor now.


Looks like it means it could get more traction usage as it gets included in this aggregator. One hopes this alleviates the pains in delivering and managing SW packages on win machines.


FUCKING YES.

Good package management (apt in particular) is a magnificent feature of Debian/Ubuntu.

The barest approximation of dependency-management in Windows is supplied by Ninite.


It's not exactly a package manager (it doesn't do proper dependency management) but for development I absolutely love Scoop[1]. It makes it so simple to get tools for Windows command line without Cygwin.

Disclaimer: I'm a contributer

[1]: http://scoop.sh/


But where will the software come from? Do you have to hook up your credit card to the package manager?


Well, one small step.

I first encountered package management in SGI IRIX. IRIX had a "mini root", which was an installer, subset of IRIX, running single user mode. Just know it's there for now, and hold that thought.

Most of the time, people launched "swmgr", which put a nice GUI on top of "inst", which was the actual tool doing the work. Inst eventually could pull from an http repository, and of course, worked with file based ones, mounted however made best sense.

That thing was seriously potent! One time, I needed to setup Alias renderer software on a bunch of machines to push out a little movie. So I just did it, remote displaying the gui on my O2 after mounting a shared image to work from.

Asked them all to load it, and about half reported out of space. Right MID INSTALL, I was given the option to do some removals, so I did, at which point the primary task could continue. Short story was it took about 30 minutes to blast some renderer software onto about 10 IRIX boxes, all with users logged in, doing stuff.

(Yes, I looked at what they were running, and pulled some sub-systems that wouldn't impact them, and no, they didn't ever know it was done, but for some disk activity)

After authoring a little script to push frames around, I set 'em all to render with every free cycle, and as users left the building, added a renderer process and escalated the priority of both. It all got done, I pulled the software, and on the machines that I had pruned from, put that all back, and never heard a peep.

Awesome. No reboots.

The other notable thing was taking my very first IRIX filesystem from a modest Indy, all the way through to a multi-CPU Origin system. Each time, I would clone it, setup on a new box, launch that mini-root, have it evaluate what was there, issue a couple of commands to keep configuration, update system, go. And it would update all the libraries, drivers, and assorted bits to run on the new box.

From time to time, I would need to do a little cleanup, maybe touching an unmanaged application and a setting or two, but I really did carry one file system through about 10 years of computing, using a package manager to keep it sorted with basically zero issues.

This was mid 90's.

Linux has most of those features today, (hello apt get!) and it's a bigger mess of packages, but then again, there is a whole lot more managed now than IRIX had. Some companies would produce managed packages and sync up with SGI, and those were great. Others were tarballs. Oh well. One could make packages and go that way, and I did a time or two, but mostly didn't.

The thing I liked the most about that era of software, and about IRIX in particular, was how just about every single thing was written to be effective and useful on the command line, and in scripted form.

Additionally, most everything had a --gui, or --verbose option, or both that would improve on the console I/O to the point where a gui could just be a wrapper, able to do what it needs to do without there being anything separate needed. When in single user mode, or remote console, terminal, whatever, really didn't matter. One could script, run a gui, command line, whatever, and it all just worked fine.

Back then, I saw Linux growing, and also later, saw Linux start to get the better bits of IRIX as SGI moved off IRIX and MIPS and onto Linux Itanium. That caused me to pick up a copy of Redhat 5.2 and begin switching over. One of the best moves I ever made actually.

So those days on IRIX are long gone, but I really do miss the fantastic systems engineering, package management, and documentation they shipped. It was complete, and if you went digging, what you needed to know actually shipped on the box with very, very few exceptions.

Windows... Yeah, it's better now. I run it all the time, and that's due to some software and the niche I'm in. No worries. But it really never did compare to the work SGI did in the 90's. Few things have.

I'm glad to see this. Just like I'm glad to see lots of things in Microsoft land, many way overdue.

I just wish it would launch a bit more complete and be a bit less painful before it really settles in and works well.

Maybe somebody should give these guys a tour of computing outside the bubble. There were a lot of really great things done. Seems to me, getting them done again, now, unabashedly is the right move. No shame. Just do it.

Lots of us would be a lot happier, and we really don't care where it comes from. Just nail it for those of us who have to continue to run the OS.

Thanks.


Please give us something that doesn't require PowerShell, since PowerShell never just works. Just look at the ridiculous bullshit that you have to deal with when you want to use Chocolatey (from the Chocolatey install guide):

Note: You must have your execution policy set to unrestricted (or at least in bypass) for this to work (Set-ExecutionPolicy Unrestricted). There have been reports that RemoteSigned is enough for the install to work.


> PowerShell never just works.

That statement is a bit much. I agree the default execution policy is annoying, but can you blame MS for being a little overzealous given their history?

Some aspects of Powershell are actually pretty nice. Ever want to use a real hashmap in bash for example? Or easy JSON construction/manipulation without hacks like jq? PS has all that.


> Ever want to...

That's because Bash is used to tie together separate programs that do the work. PowerShell, on the other hand, is a .NET shell and more equivalent to the shells in Python, Node, Ruby, the JVM, etc, all of which provide ways to work with JSON and many different data structures.


That's a false distinction. bash has a (primitive, idiosyncratic) language built-in. So does PS. They also are both designed to glue together independent programs via the pipe abstraction. The difference is that PS has much more modern language constructs and capabilities. The fact that it hooks into the .NET API as well is only a convenience.


> They also are both designed to glue together independent programs via the pipe abstraction

PowerShell and its object pipeline are focused on cmdlets which are not arbitrary programs, they are .NET classes, and PowerShell objects are .NET objects.

Unix shells like sh and bash are focused on tying together separate programs and only have a very minimal set of built-in commands.

The focus of each one is very different, despite PowerShell's syntax.

Piping objects between cmdlets in PowerShell is analogous to working with objects in Python, Ruby, Perl, etc, not like Unix pipes.


> PowerShell and its object pipeline are focused on cmdlets which are not arbitrary programs, they are .NET classes, and PowerShell objects are .NET objects.

Both correct and horribly wrong.

PowerShell is indeed focused on cmdlets (and functions) and it's pipeline concept is focused on objects. It is also correct that cmdlets are not arbitrary programs.

However, cmdlets are not .NET classes. It is true that you can implement a cmdlet through a .NET class that derives from the base Cmdlet class. But that is just one way to create a cmdlet that can be recognized by PowerShell. You can build cmdlets using PowerShell itself. There is nothing in the PowerShell specification (http://www.microsoft.com/en-us/download/confirmation.aspx?id...) that require a cmdlet to be a .NET class.

If you use PowerShells (rich) introspection function to enquire about a cmdlet, there is nothing objective to give it away as either .NET class based or advanced function based (advanced functions are cmdlets). Yes, you can see if a module (distribution unit of cmdlets, providers, functions etc) are "binary" or "script". That merely reinforces the argument that cmdlets are not classes.

Indeed, PowerShell will now dynamically create a new type of cmdlets based on an Open Management Infrastructure (OMI) implementation. Think managed routers, switches, Linux boxes etc. If they expose a REST service that complies with OMI, PowerShell can query the capabilities and create cmdlets in-session for each individual type of equipment. PowerShell has since v3 been able to create WMI/CIM cmdlets.

The point is that your assertion that cmdlets are .NET classes is both factually wrong, and grossly misleading. Saying that PowerShell cmdlets are .NET classes is like saying that Yum packages are just archives. Sure, you can pick one apart and say "look! it's a .NET class". I can pick a Yum package apart and say: "look! it's just an archive with some files in it!".

That said, it is true that PowerShell builds upon .NET. It is implemented using .NET and the object model is based on .NET. PowerShell's type system is a superset of .NETs - because an administrators tool needs more ad-hoc typing.

We have gotten so used to think that sh shells were developed with pipelines like that (streams stdin/stdout) because the commands are like that. In reality, the commands and the (first) shells were developed in tandem so that the commands would fit the shell and vice versa.

What Microsoft is doing with PowerShell is challenge that commands must be "like that". If we make the commands "richer" and allow them to use a higher level protocol (objects) for inter-command communication, an even smarter shell can be built. Yes, it does create some friction with traditional commands. PowerShells solution is to regard those as "cmdlets" that just consume sequences of strings (which are objects) and produce sequences of strings.

> Unix shells like sh and bash are focused on tying together separate programs and only have a very minimal set of built-in commands.

PowerShell is focused on tying together separate cmdlets and only has a very minimal set of build-on commands. See?

(Aside: Actually, PowerShell is more terse than any sh shells: It has 0 (zero) "built-in" commands. What it is has is a module concept with auto-discovery. The seemingly "built-in" are not built in. Those cmdlets comes from modules that are distributed alongside the PowerShell binary, and PowerShell auto-loads them as it will any module that is placed in the correct location. sh shells needs commands like e.g. "cd" that manipulate the environment to be built-in. You could never implement "cd" as an external command. In PowerShell "cd" (alias for the Set-Location cmdlet) is distributed through the Microsoft.PowerShell.Management module)

> The focus of each one is very different, despite PowerShell's syntax. Piping objects between cmdlets in PowerShell is analogous to working with objects in Python, Ruby, Perl, etc, not like Unix pipes.

No it is not. A pipeline is exactly that: A temporary communications channel that is set up for the duration of the (compound) command, and through which multiple objects flow. In PowerShell the individual cmdlets - like commands in a sh pipeline - are active a the same time. From the PowerShell spec: "A pipeline is a series of one or more commands each separated by the pipe operator | (U+007C). Each command receives input from its predecessor and writes output to its successor."

Re: Python, Ruby, Perl, etc: You can create a pipeline concept for those languages as well. See for instance pypi (https://code.google.com/p/python-pipelines/). But it is like you want to portrait PowerShell pipelines as mere method invocations. They are considerably more.


> your assertion that cmdlets are .NET classes is both factually wrong

They are instances of .NET classes that derive from `Cmdlet` and `PSCmdlet` and run in the PowerShell runtime. As stated plain as day in the Microsoft cmdlet documentation:

"Cmdlets are instances of .NET Framework classes; they are not stand-alone executables."

Your entire comment is an attempt to obfuscate the nature of PowerShell cmdlets, which are actually very simple.

>> Piping objects between cmdlets in PowerShell is analogous to working with objects in Python, Ruby, Perl, etc, not like Unix pipes.

> No it is not.

Yes it is.

This is not about the term 'pipeline,' it's about how PowerShell, Unix shells and pipes, and language runtimes work and the scope to which they apply. FACT: PowerShell's object pipeline is internal to the PowerShell / .NET runtime, just as other language features of Python, Ruby, etc are internal to those languages' runtimes. FACT: Unix pipes are primarily for chaining separate processes. They are a lower-level feature than the runtimes of Python, Ruby, or PowerShell, and can be used to chain those runtimes together with each other and other processes.

Here's an illustration of the scope of each one.

PowerShell's cmdlet / object pipeline:

    (PowerShell runtime)
Unix Pipeline:

    (Python runtime) | (arbitrary executable) | (PowerShell runtime) | (Ruby runtime)


You look at the developer documentation for developing PowerShell cmdlets using C#, and go "look! it says you create cmdlets using C#!".

Well, here is another way to create cmdlets: https://msdn.microsoft.com/en-us/library/jj542520(v=vs.85).a...

Yes, you specify cmdlets using XML. You can point to static methods, remote CIM objects.

It may very well use .NET classes underneath. The implementation details are immaterial to how PowerShell works.

PowerShell is certainly built using .NET - and require that commands are built (or at least execute) under .NET.

However, PowerShell works with objects. An object is a collection of properties and methods that work on the object. Every command in PowerShell must be able to understand the objects, hence a common binary object format must be specified. PowerShell choses .NET objects as it's lingua franca - the PowerShell type system, although it extends it.

PowerShell specification says that commands that specifies parameters must do so with types from that type system, and if they support pipelines they must consume objects from the type system and produce objects from the type system. It is a protocol

This is very much like how sh shells specifies that commands must support arguments as an array of strings and can optionally consume byte streams from stdin and produce byte streams to stdout. It's a protocol

Does PowerShell use the same protocol as sh shells? No. Hence, the commands are different.

You do not like that they are different? fine. Don't use it.

Your pipeline illustration should have been:

PowerShell's cmdlet / object pipeline: (PowerShell runtime | adapter(arbitraty old-school stdin/stdout) | ironpython runtime | any runtime that adheres to type system)

Unix Pipeline: (Python runtime) | (arbitrary executable) | (PowerShell runtime) | (Ruby runtime)

Yes, PowerShell raises the bar for commands: They must meet some other criteria than string args and stdin/stdout. But just like sh shells and stdin/stdout executables, PowerShell and cmdlets build an ecosystem. It may hurt to realize that the world one knows is just one possible world, and that there may be others out there where some things may be easier and other harder.

To me, it is an opportunity to learn. Nothing quite like being shaken up a little.


What still seems to be getting missed is that despite some superficial similarities, Unix pipelines and PowerShell pipelines serve different purposes, operate on different levels, function totally differently, and are not interchangeable within the system.

Unix pipelines are an operating system feature that coordinates data streams between processes. PowerShell's object pipeline is function chaining feature of PowerShell, a runtime that runs on an operating system as a process, the kind of process whose communication with other processes is coordinated on Unix systems using Unix pipelines.

Python, PowerShell, and Ruby object handling and function/method chaining are not a features at the operating system level, they are features within the Python, PowerShell, and Ruby runtimes and processes. Unix pipelines are a feature at the operating system level that coordinate communication between these kinds of processes.

Unix pipelines are not a feature of Unix shells, they are an operating system feature for which shells provide an interface. PowerShell's object pipelines are features of PowerShell and its runtime, not the host operating system.

This is not a minor distinction. A feature for working with objects and chained functions within particular language or framework's runtime is not the same thing as a feature for coordinating process communication in an operating system.


I disagree, Powershell is basically a way to script .Net. It has provider so you're able to hook into functionality throughout the OS, but Powershell really wants to be a first class .Net language (and it is).


So?


If you want to do advanced stuff in your scripts, you've never been limited on unix systems to just sh or bash, but on Windows you still don't have a good shell for interactive use.


How is it fair to give the example of the Python shell in Unix but not in Windows when it runs in both?


What? Python isn't the kind of thing you would use as an everyday interactive system shell; it's a scripting language that can incidentally be used interactively but isn't well-suited for that, especially without extras like ipython. And quite aside from that, every unix-like desktop OS comes with Python pre-installed; Windows doesn't.


PowerShell ISE is pretty nice. I use it as my main shell.


So it's not unique to Windows and other things have been doing this for much longer on different platforms with more users and now have much larger ecosystems making it easier to do what you want quickly.


I have had so many people tell me how fantastic PowerShell is. Not a single one of them was ever a *nix user. PowerShell is getting better with a tab completion of sorts but it is still a headache to figure out commands without web search. I try to force myself to use PowerShell as often as possible, but continuously find myself going back to a GUI. I have the opposite problem when using Linux.


I think this comes from being unfamiliar with it.

Want to know what you can do with network adapters? Run Get-Help AD to see every cmdlet containing AD, and narrow your search from there.

Get-Help auto-updates itself. See a command, but want to know all of the ins and outs (syntax etc) of using it? Run Get-Help [command]. Use the optional -Examples flag to see several well-explained examples. You can even add -Online to get the technet article to launch in your default web browser.

I would think that nix users would feel at home with powershell. Microsoft has made a big push for getting developers to take a second look at Windows. They even set up aliases for common nix commands. Don't type 'dir', just type 'ls' - It works fine.


> Microsoft has made a big push for getting developers to take a second look at Windows.

This presumes that the developers' first look and decision to stay away from Windows was mistaken. It wasn't.

I avoid Windows because mistakes were made. If Microsoft continues to assert that I'm the one who made the mistakes, then I'm just going to keep avoiding their stuff.

As a developer, I make dependency choices based largely on my confidence in the upstream's judgment. Microsoft has a long track record of making poor technical decisions and then never fixing them. I've also heard enough about their internal culture to understand how difficult this would be to change. Introducing a shell or a package manager well over a decade after it was clear that they needed one doesn't inspire confidence.

Not to mention Microsoft's tendency act coercively when they can get away with it, and friendly when they can't. Yes, I'm bitter about this.

I still don't plan on taking a second look.


Yeah MS was burned by scripting in the past (vbs attachments). So now they ship their new server scripting system... disabled by default.

Same type of reasoning that leads them to not include a telnet client. Or makes you twice opt in to compression to make IIS actually gzip "dynamic content". It's a noble idea, done more in a CYA kind of way, it feels.


PowerShell is disabled by default for security reasons. It's one command to enable it.


> PowerShell is disabled by default for security reasons.

Small correction: PowerShell script files are disabled by default for security reasons.

PowerShell is available with all Windows versions since Vista SP1.

On the latest Windows versions, remote command (part of Remote Management) is enabled by default. I.e. you can connect (given proper credentials) to a Server 2012R2 through PowerShell remoting and execute commands.

Indeed, the new "Server Nano" will not have any way to log in locally at all. There is no shell (as in operating system shell), and no local editor (like notepad). Only the remote command interface. PowerShell will be extended to allow remote file editing - i.e. use a local editor for a file residing on the remote computer (been missing that for the longest time now)


Yeah, one command - on every machine... Meanwhile on Linux, I don't need to do any special configuration to run scripts.


You do if those scripts can affect the system and need to run as root. Powershell is incredibly powerful (and dangerous) and some security isn't the worst thing.


the irony of this is that if ms were to leave powershell open to run any scripts from anywhere at any time, they'd be ridiculed for lax security policies.


why? .cmd and .bat can run any time, and can do just as much damage as powershell.


It's no more powerful and dangerous than bash in the right hands.


> Yeah, one command - on every machine...

I think Microsoft envisions centrally-managed environments, rather than those with lots of Windows machines that aren't centrally managed.


> on every machine...

Or through one group policy pushed to every machine.


Plus, unlike what most people envision on here, Microsoft intends companies to push AllSigned or RemoteSigned, rather than Unrestricted.

So the company can push their CA and new PS policy in a single GPO, and then all internal PS scripts are signed using an internal CA generated code signing certificate.

This sounds complex but it is actually as simple as running Set-AuthenticodeSignature on each script using the code signing certificate.


It's like they wanted sudo and user accounts with arbitrary permissions but had no idea how to do that, so they did UAC and this stuff instead.


Windows has an analogy to sudo, called runas

UAC is designed to solve a different problem, users on Windows historically ran as administrator, and much legacy software requires it, UAC is designed to provide an apparent administrator environment to the software but catch what the software is using, and give the user a chance to approve privilege escalation.


runas is more of a su analogy than a sudo one.


I disagree, the -u option for sudo gives you nearly identical functionality.


The reason su is a better runas analogy than sudo is that it has a similar scope and similar purpose.

While it does overlap su/runas functionality, sudo has a much larger superset of features and policy configuration that su and runas simply don't have.


how was the saying, "Those who do not understand UNIX are condemned to reinvent it, poorly"..


Bash is oudated. PowerShell is new.


I wrote to m$ requesting this feature a few years ago, happy it got implemented regardless of they listened or not. It will be good for security and admins/devops.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: