San Francisco Museum of Modern Art

Stand on Third Street for a moment and look up. A wall of white rises behind the older brick building, rippled like water or a bank of fog. As the light moves across the city, that surface turns pale grey, then cream, then almost silver. Traffic hums behind you. Somewhere a cable is clanking, a bus is braking. And in front of you sits one of the largest museums of modern and contemporary art in the United States.

The San Francisco Museum of Modern Art began in 1935. It was the first museum on the West Coast devoted to modern and contemporary art. That was a bold idea at the time. Modern art was still new and divisive, and many people found it confusing or even offensive. The museum first opened under a slightly different name, inside the War Memorial Veterans Building in the city's Civic Center. For its first six decades, it shared space with other institutions.

The museum moved to this South of Market neighbourhood in 1995. Back then the area was mostly warehouses, parking lots and small workshops. The move was a gamble on a district that many locals avoided after dark. Today the streets around you hold galleries, hotels, offices and the green lawns of Yerba Buena Gardens across the road. The museum helped to start that change.

Look at the older building at the front. This is the work of the Swiss architect Mario Botta. Notice the warm red brick laid in patterns, and the stepped, almost fortress-like shape. On top sits a large cylinder cut at an angle, banded in black and white stone. That cylinder is a skylight. It pulls daylight deep down into the centre of the building. Many visitors walk straight past without looking up at it from inside — so make a note.

Now look at the white wall rising behind it. This is the expansion by the Norwegian firm Snøhetta, which opened in 2016. The architects took inspiration from the fog and water of San Francisco Bay. The panels are made from a fibre-reinforced polymer, a light, strong material often used in boat building. Here is a detail that most people miss. Tiny crystals of silica sand from Monterey Bay are set into the surface. That is why the facade catches light differently at every hour.

The expansion roughly tripled the museum's gallery space. That gave room for a gift that changed the museum. Doris and Donald Fisher, the couple who founded the clothing company Gap, collected art for decades. Their collection of postwar and contemporary art came to the museum through a long-term partnership. It includes large groups of works by artists such as Alexander Calder, Ellsworth Kelly, Gerhard Richter and Andy Warhol. Few private collections of this kind live inside a public museum.

When you enter, pay attention to the ground floor. In the large gallery near the Howard Street side, you may find a sculpture by Richard Serra called Sequence. It is made of huge curving walls of weathering steel, rusted to a deep brown. You do not look at this work so much as walk through it. The walls lean in and out as you move. Your sense of balance shifts. Your voice sounds different inside the curves.

As you climb, notice the main staircase. It is made of pale maple wood, and its wide steps are built for sitting as well as walking. People pause here to rest, chat or watch the crowd. It feels more like a public square than a corridor. This is a deliberate choice. The architects wanted the museum to feel open to the city, not closed off from it.

The collection covers painting, sculpture, design, architecture, media art and photography. The museum has collected photographs since its early years, long before many American museums took photography seriously as an art form. Today it has a large centre dedicated to photography. If you love cameras and images, give yourself extra time there. The galleries show everything from early prints to contemporary work.

Here is a second surprise. Among the museum's well-known works is a piece by Robert Rauschenberg called Erased de Kooning Drawing, from 1953. The young Rauschenberg asked the older painter Willem de Kooning for one of his drawings. Then he spent weeks rubbing it out with erasers. What remains is a faint smudge on paper, framed and labelled. Is it destruction, or a new work of art? That question is the whole point.

The museum also holds paintings that changed the story of modern art. Henri Matisse's Woman with a Hat, from 1905, caused an uproar when it was first shown in Paris. Critics mocked its wild, unnatural colours. Frida Kahlo's double portrait of herself with her husband Diego Rivera was painted in San Francisco in 1931. Look for it, and you may notice how small she paints herself beside him. Works are rotated and loaned, so not every piece is on view at all times.

Take time for the outdoor terraces on the upper floors. One holds a large living wall, covered in plants that grow straight up the side of the building. On a clear day, the air smells of damp leaves and soil. On a foggy day, the plants drip and the city sounds soften. These terraces offer a quiet break between galleries. They also show you how the museum sits among the towers of downtown, framed by glass and sky.

Before you leave, step back out onto the pavement and look once more. The red brick of 1995 and the white ripples of 2016 stand side by side. Two very different ideas about what a museum should be... and somehow they work together. As you walk away towards Yerba Buena Gardens or Market Street, notice the light on that white wall. It will not look the same as when you arrived.