I know the original brioche recipe is supposed to be slightly sweet, but I really enjoy a brioche hamburger bun that one of my local bakeries does and it's really savoury. I tried making brioche several times now, and the maximum amount of salt that I used was 7g (for like 340g of flour), but it ended more sweet than savoury every time. I know this may sound like a simple answer: just add more salt, but I'm afraid of doing so because I know salt negatively affects yeast development. Can I just add more salt until I like it and it won't affect the dough rising so much, or maybe should I remove some sugar? The recipe I follow calls for 35g sugar (again considering 340g flour) plus 60g milk which is already slightly sweet. 
Brioche is not necessarily sweet.  By definition it is a bread with a high butter and egg content...an enriched dough.  Also, salt does not necessarily = savory.  Umami defines savoriness. Fermentation increases umami.  It is one of the reasons we enjoy bread in the first place.  Adding more salt will just make your recipe more salty.  I would consider reducing the sugar, and consider the quality of the milk, butter, and flour you are using. 