John Langford and the team (Microsoft Research) have built a contextual bandit library in Vowpal Wabbit.

It can be used from active learning to changing webpage layouts to increase ad clicks. It has the best bounds out of all exploratory algorithms.

Structured contextual bandits that come with LOLS (another algorithm present in vowpal wabbit) is extremely powerful.

All for free under BSD3.