Size bidisperse granular materials in a spherical tumbler segregate into two different patterns of three bands with either small particles at the equator and large particles at the poles or vice versa, depending upon the fill level in the tumbler. Here we use discrete element method (DEM) simulations with supporting qualitative experiments to explore the effect of the tumbler wall roughness on the segregation pattern, modeling the tumbler walls as either a closely packed monolayer of fixed particles resulting in a rough wall, or as a geometrically smooth wall. Even though the tumbler wall is in contact with the flowing layer only at its periphery, the impact of wall roughness is profound. Smooth walls tend toward a small-large-small (SLS) band pattern at the pole-equator-pole at all but the highest fill fractions; rough walls tend toward a large-small-large (LSL) band pattern at all but the lowest fill fractions. This comes about because smooth walls induce poleward axial drift of small particles and an equator-directed drift for large particles, resulting in an SLS band pattern. On the other hand, rough walls result in both sizes of particles moving poleward at the surface of the flow, but due to radial segregation, small particles percolate lower in the flowing layer where there is a return drift toward the equator while large particles remain at the surface near the pole, resulting in an LSL band pattern. The tendency toward either of the two band patterns depends on the fill level in the tumbler and the roughness of the tumbler's bounding wall.