title: Python - A way to learn and detect text patterns?
body: <p><strong>Problem:</strong></p>
  
  <p>I am given a long list of various position titles for jobs in the IT industry (support or development); I need to automatically categorize them based on the general type of job they represent. For example, IT-support analyst, help desk analyst... etc. Could all belong to the group <em>IT-Support</em>.</p>
  
  <p><strong>Current Approach:</strong></p>
  
  <p>Currently, I am manually building regex patterns to accomplish this, which change as I encounter new titles which should be included in a group. For example, I originally used the pattern:</p>
  
  <p><em>"(HELP|SERVICE) DESK"</em></p>
  
  <p>to match <em>IT-Support</em> type jobs, and this eventually became:</p>
  
  <p><em>"(HELP|SUPPORT|SERVICE) (DESK|ANALYST)"</em></p>
  
  <p>which was even more inclusive.</p>
  
  <p><strong>Question:</strong></p>
  
  <p>I feel like there should be a fairly intuitive way to automatically build these regex patterns with some sort of algorithm, but I have no idea how this might work... I've read about NLP briefly in the past, but its extremely alien to me... Any suggestions on how I might implement such an algorithm with/without NLP?</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>I'm considering using a decision tree, but it has some limitations which prevent it from working (in this situation) "out-of-the-box"; for example, if I have built the following tree: </p>
  
  <p><strong>(Service)->(Desk)->(Support) OR ->(Analyst)</strong> <em>...where Support and Analyst are both children of Desk</em></p>
  
  <p>Say I get the string "Level-1 Service Desk Analyst"... This should be categorized using the decision tree above, but it will not inherantly match the tree (since there is no root node named "Level" or "Level-1").</p>
  
  <p>I believe I am heading in the right direction now, but I need additional logic. For example, if I am given the following hypothetical strings:</p>
  
  <ol>
  <li>IT Service Desk Analyst</li>
  <li>Level-1 Help Desk Analyst</li>
  <li>Computer Service Desk Support</li>
  </ol>
  
  <p>I would like my algorithm to create something like below:</p>
  
  <p><strong>(Service OR Help)->(Desk)->(Analyst OR Support)</strong> <em>...where Service and Help are both root nodes, and both Analyst and Support are children of Desk</em></p>
  
  <p>Basically, I need the following: <strong>I would like this matching algorithm to be able to reduce the strings it is presented with to a minimal number of sub-strings which effectively match all of the strings in a given cateogory (preferably using a decision tree)</strong>.</p>
  
  <p>If I am not being clear enough, just let me know! </p>