Nyholms research in inorganic chemistry was primarily concerned with the preparation of transition metal compounds, particularly those involving organo-arsenic ligands.  His interest in organoarsenic chemistry was fostered at the University of Sydney by George Joseph Burrows (1888–1950).  Using the strong chelating ligand diars, Nyholm demonstrated a range of oxidation states and coordination numbers for several of the transition metals. Nyholm noted that the term unusual valence state had an historical, but not chemical significance.  The definition of usual oxidation state refers to oxidation states that are stable in environments made up of those chemical species that were common in classical inorganic compounds, e.g. oxides, water and other simple oxygen donors, the halogens, excluding fluorine, and sulphur.  Nowadays, however, such species constitute only a minority of the vast number of donor atoms and ligands that can be attached to metal.'
After joining Sydney Technology college in 1940 Nyholm formed a close personal friendship with Francis (Franky) Dwyer and they collaborated in their research. Despite heavy teaching loads, between 1942 and 1947 they reported complexes of rhodium, iridium, and osmium in seventeen papers in the Journal and Proceedings of the Royal Society of New South Wales.
One of Nyholm's early successes was the preparation of an octahedral complex of trivalent nickel [Ni(diars)Cl]Cl, by aerial oxidation of the red salt of bivalent nickel [Ni(diars)]Cl. He also described stable complexes of quadrivalent nickel such as the deep blue [Ni(diars)Cl] [ClO], by nitric acid oxidation of the trivalent complex. This stabilisation of higher oxidation states became significant in the Nyholm-Rail reaction where the ditertiary arsine, diars undergoes a condensation reaction to a tritertiary arsine, triars.  Nyholm prepared examples of divalent octahedral complexes of the type M(diars)X, where X is Cl, Br or I, and M is Cr, Mn, Fe, Co, Ni, Mo, Tc, Ru, Pd, W, Re, Os, and Pt.
Many of these divalent complexes are sensitive to aerial oxidation.  The chromium complex is oxidized by water.  Indeed, previous attempts to prepare Cr(diars)X had failed.  The chromium compounds were eventually synthesized by his co-worker Anthony Nicholl Rail only a month before Nyholm's death, using rigorous air-free techniques.
Together with Professor Ronald Gillespie, Nyholm developed the VSEPR (Valence shell electron pair repulsion) theory for the simple prediction of molecular geometry. This theory emphasized classical pictures of bonding, adapted to include features of quantum theory, but focusing on electron clouds of varying density within a probability envelope.