The flow driven by a rotating disk of a thin fluid layer in a fixed cylindrical casing is studied by direct numerical simulation and experimental flow visualizations. The characteristics of the flow are first briefly discussed but the focus of this work is to understand the transition to the primary instability. The primary bifurcation is 3D and appears as spectacular sharp-cornered polygonal patterns located along the shroud. The stability diagram is established experimentally in a (Re, G) plane, where G is the aspect ratio of the cavity and Re the rotational Reynolds number and confirmed numerically. The number of vortices scales well with the Ekman number based on the water depth, which confirms the existence of a Stewartsonlayer along the external cylinder. The critical mixed Reynolds number is found to be constant as in other rotating flows involving a shear-layer instability. Hysteresis cycles are observed highlighting the importance of the spin-up and spin-down processes. In some particular cases, a crossflow instability appears under the form of high azimuthal wave number spiral patterns, similar to those observed in a rotor-stator cavity with throughflow and coexists with the polygons. The DNS calculations confirm the experimental results under the flat free surface hypothesis.