Mental Health Assistant RAG

.envrc

@@ -0,0 +1,2 @@
+export GROQ_API_KEY="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"
+export MISTRAL_API_KEY="8liS0F7zb7zNgujwgo3dgTbr6HbfvmeI"

.gitignore

@@ -0,0 +1 @@
+.env

Dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.12-slim
+
+WORKDIR /app
+
+RUN pip install pipenv
+
+COPY dataset/data.csv dataset/data.csv
+COPY ["Pipfile", "Pipfile.lock", "./"]
+
+RUN pipenv install --deploy --ignore-pipfile --system
+
+COPY src .
+
+EXPOSE 8501
+
+CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.enableCORS=false"]

Pipfile

@@ -0,0 +1,22 @@
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[packages]
+mistralai = "*"
+matplotlib = "*"
+seaborn = "*"
+streamlit = "*"
+scikit-learn = "*"
+groq = "*"
+python-dotenv = "*"
+tqdm = "*"
+psycopg2-binary = "*"
+gunicorn = "*"
+plotly = "*"
+
+[dev-packages]
+
+[requires]
+python_version = "3.9"

dataset/ground_truth_data.csv

@@ -0,0 +1,367 @@
+id,question
+1590140,How do mental illnesses affect a person's daily functioning and relationships?
+1590140,What are some examples of serious mental illnesses that can result in major functional impairment?
+1590140,"Why is it a misconception to associate mental illness with a person's character, intelligence, or willpower?"
+1590140,"How are mental illnesses treated, and what is the success rate of these treatments?"
+1590140,Can a person with mental illness become independent and successful with proper treatment?
+2110618,Who are the groups most commonly affected by different mental illnesses?
+2110618,How do mental illnesses affect children and the elderly?
+2110618,What behaviors should parents and caretakers watch out for in children that may indicate a serious emotional disturbance?
+6361820,How does mental illness affect different populations?
+6361820,At what age are individuals most susceptible to mental illness?
+6361820,What signs should parents and caretakers look out for in children under 18 that may indicate a mental health condition?
+9434130,What are some general symptoms that may indicate a mental health disorder in adults?
+9434130,Can you name some symptoms that could suggest a mental health issue in older children and pre-teens?
+9434130,Are there any signs that might point to mental health problems in younger children?
+9434130,What kind of behavior in children might suggest they are struggling with mental health?
+9434130,What are some potential indicators of mental health issues related to mood or eating habits?
+7657263,Can early identification and treatment improve recovery from mental illness?
+7657263,Are there effective treatments available for different types of mental illnesses?
+7657263,What is the importance of a person's active engagement in their own recovery process from mental illness?
+7657263,Is it common for some people with mental illness to experience a return of symptoms after treatment?
+7657263,How can one live a fulfilled and productive life despite careful monitoring and management of mental illness?
+1619387,What should I do if I suspect someone I know has a mental disorder?
+1619387,How can I help a friend or family member who might be experiencing symptoms of a mental health condition?
+1619387,What are the benefits of speaking with a mental health professional if I think a loved one has symptoms of a mental disorder?
+1030153,How can I search for mental health providers in my area?
+1030153,What should I consider when researching mental health professionals for myself or my child?
+1030153,How can I ensure that I will feel comfortable with the mental health professional I choose?
+8022026,What types of treatments are available for mental illness?
+8022026,How do different treatment options work for different people?
+8022026,How can I find the most effective treatment for myself or my child?
+1155199,What can I expect in terms of the cause of my illness?
+1155199,Are there other conditions that share my symptoms?
+1155199,What kind of physical or neurological tests are usually involved?
+1155199,What other specialists might be involved in my treatment and who will coordinate?
+1155199,What is the typical role of the family in the treatment process?
+7760466,What are the differences in education and training among mental health professionals?
+7760466,How do mental health professionals specialize in different areas?
+7760466,How can I find a mental health professional that fits my specific needs?
+2553795,What steps can I take to find a mental health professional who is a good fit for me or my child?
+2553795,How can I ensure that my child or I will feel comfortable working with a mental health professional?
+2553795,What resources are available to help me research and find the right mental health professional for my needs or my child's needs?
+1259439,What can you tell me about the cause of my illness?
+1259439,Are there other conditions that share my symptoms?
+1259439,What kinds of examinations do you usually conduct?
+1259439,What is your recommended treatment program for my diagnosis?
+1259439,What can I expect from the medications prescribed for my condition?
+4197817,Where can I find mental health resources in my area?
+4197817,What kind of organizations offer support groups for consumers and their families?
+4197817,Are support groups always led by mental health professionals or can they be peer led?
+3388962,What is the primary purpose of the new medication?
+3388962,How long does it typically take for the medication to take effect and how will I know when it is effective?
+3388962,"What are the proper directions for taking this medication, including any food, drinks, or activities to avoid?"
+3388962,What side effects should I be aware of and what should I do if they occur?
+3388962,What is the procedure if I accidentally miss a dose of the medication?
+5343181,"If I feel better after taking medication, does that mean I can stop taking it without consulting my doctor?"
+5343181,What might happen if I stop taking my medication abruptly?
+5343181,How can I address side effects that make me unwilling to continue treatment?
+5778437,What organizations provide prescription assistance programs for individuals with financial needs?
+5778437,Do I need a doctor's consent to apply for prescription assistance programs?
+5778437,What kind of proof do I need to provide for these prescription assistance programs?
+9541219,Where can I find a therapist who specializes in my specific mental health condition?
+9541219,Are there different types of therapy for children and adults?
+9541219,What kind of therapy would be most effective for my symptoms?
+3268009,What are some alternative types of mental health treatment?
+3268009,Are there any self-help plans for mental health conditions?
+3268009,What is the role of peer support in mental health treatment?
+3340726,What are the different kinds of mental health professionals that I can choose from?
+3340726,How can I find the right mental health professional for my needs?
+3340726,Can you give me more information about the various types of mental health professionals?
+9539480,Where can I find peer support organizations?
+9539480,Are there support groups led by mental health professionals?
+9539480,How can I find support groups for my family and friends?
+4211025,Where can I find help for a mental health crisis?
+4211025,What kinds of services does inpatient care provide for people in crisis?
+4211025,How can inpatient care assist with adjusting to new medications or symptoms?
+4031860,Where can I find resources to help me locate mental health therapists in my area?
+4031860,How can I learn about different mental health treatment options in my community?
+4031860,Can you suggest any ways I can better understand the treatment process for mental health services?
+5215843,What are the risks associated with clinical trials?
+5215843,How can I find out about new treatment approaches being tested?
+5215843,Where can I get more information about participating in research studies?
+8927672,What is a psychiatric advance directive and how does it differ from a medical advance directive?
+8927672,How can I create a psychiatric advance directive and who will it impact during a mental health crisis?
+8927672,What kind of treatment instructions can I include in a psychiatric advance directive?
+7728364,What exactly makes up mental health?
+7728364,Can you explain the components of mental health?
+7728364,"How are beliefs, thoughts, feelings, and behaviors related to mental health?"
+4194958,What can I do if the usual support methods don't work for me?
+4194958,How can I stay hopeful while searching for the right help?
+4194958,What are some other approaches I can try if the current ones don't help?
+1667863,Is it possible to take steps to avoid mental health issues?
+1667863,How does working on our wellbeing and resilience contribute to preventing mental health problems?
+1667863,What role does seeking help early play in preventing mental health issues from becoming severe?
+8904276,Are there any treatments that can help with mental health issues?
+8904276,Is it possible to find relief from mental health problems through talking and counselling?
+8904276,"Can medication, friendships, exercise, sleep, and nutrition play a role in addressing mental health concerns?"
+4283807,What are some psychological causes of mental health problems?
+4283807,What role does biology play in mental health issues?
+4283807,How can social factors contribute to mental health problems?
+8690253,Who are some people I can talk to if I'm worried about my mental health?
+8690253,What steps can I take to find out more information about what I'm experiencing with my mental health?
+8690253,How can talking to someone about my mental health help me get perspective and start getting help?
+8690253,What kind of help can I expect to receive after I talk to someone about my mental health?
+8690253,What can I do to get started on the path of improving my mental health?
+2973656,How can I tell if my beliefs and thoughts are affecting my daily functioning?
+2973656,What are the signs that my feelings and behaviors are having a significant impact on my ability to function normally?
+2973656,When is it important to seek help if my mental health is affecting my daily life?
+4759773,What actions can I take if I'm concerned about a loved one's mental health?
+4759773,How can I approach a friend or relative about seeking mental health support?
+4759773,What is the first step I should take when I notice a concern for someone's mental well-being?
+1511075,How should I handle people giving me advice based on evidence?
+1511075,What is the best way to find out what works for me when people tell me what to do?
+1511075,How can I balance following others' advice and doing what is best for my mental health?
+7069853,Why is it important for my mental health to stay informed about the pandemic?
+7069853,How can reliable and up-to-date information about the coronavirus help me feel more resilient?
+7069853,What are the negative effects of following the news all day and how can I avoid them?
+4181750,How can I determine if a source is reliable for information about the coronavirus and COVID-19?
+4181750,What should I consider when evaluating the credibility of experts quoted or cited in health articles?
+4181750,How can I identify if a source is trying to make me feel a certain way rather than providing factual information?
+4181750,What is the role of social media in obtaining reliable health information and what cautions should I take?
+4181750,Where can I learn more about digital and media literacy and thinking critically around health claims?
+2554141,Why does having a plan help manage fear and stress during a pandemic?
+2554141,What are some tasks I can include in my daily schedule for managing during a pandemic?
+2554141,Where can I find resources on problem-solving to help me manage during a pandemic?
+2554141,How can I practice problem-solving to manage uncertain situations during a pandemic?
+2554141,What are some ways I can manage feelings of being overwhelmed or hopeless during a pandemic?
+9100298,What are some ways I can maintain social connections while physically separated from loved ones?
+9100298,What should I do if I'm feeling lonely during this time?
+9100298,Where can I find local community support groups and mutual aid groups online?
+2884887,How does taking care of my physical health impact my mental health?
+2884887,What are some tips for maintaining a balanced diet while spending more time at home?
+2884887,What are some ways to stay active and exercise while staying safe at home?
+2884887,How can I practice healthy sleep habits if stress and anxiety are affecting my sleep?
+2884887,What should I do if I have a chronic illness and need to take additional precautions during this time?
+9250044,What are some low-effort distractions I can try when I'm feeling overwhelmed or isolated?
+9250044,What are some ways I can keep myself engaged and focused if I have more energy?
+9250044,How can I use online resources to learn new things and stay entertained during social distancing?
+9250044,What are some ways I can maintain healthier habits and avoid unhelpful ones when my schedule is disrupted?
+9250044,How can I get involved in helping my community while practicing social distancing and good hygiene?
+3597720,How does the way I think about a situation impact my feelings and behaviors?
+3597720,What can challenging negative thoughts do to improve my mood and help me see new options?
+3597720,Can you give me an example of how to reframe a negative thought to a more positive or productive one?
+3597720,How can I better understand the situation more realistically and see the parts that I can control right now?
+3597720,"What can I do to focus on the things I can do, and make everything feel more manageable?"
+7535002,What strategies can I use to challenge common thinking traps?
+7535002,How can I identify if I'm falling into distorted thinking patterns?
+7535002,What is a helpful way to break a thinking trap by looking at the thoughts?
+4134858,What are some common causes of grief outside of losing a person or pet?
+4134858,How can I healthily express my feelings of grief?
+4134858,Where can I find help from a mental health professional if I'm struggling with grief?
+9676742,How can I find a family doctor using the College of Physicians and Surgeons of BC's Find a Physician tool?
+9676742,Are walk-in clinics a good option for finding a regular doctor if I have ongoing care needs?
+9676742,How can I obtain a referral to see a psychiatrist?
+9676742,Where can I find a registered psychologist in BC?
+9676742,How can I visit the BC Association of Clinical Counsellors to find a clinical counsellor?
+2612846,Can I make an appointment with a psychiatrist directly or do I need a referral from a doctor?
+2612846,What is the process for self-referring to a mental health program in BC?
+2612846,How can I find my local mental health center in BC?
+2612846,What services are available at a mental health center besides access to a psychiatrist?
+2612846,How can I get help from HealthlinkBC if I need assistance finding a mental health service?
+3166337,Where can I find a Registered Psychologist?
+3166337,Do I need a doctor's referral to see a psychologist?
+3166337,How can I find out about lower-cost options for counselling in my community?
+3166337,Are the costs of seeing a Registered Psychologist covered by MSP?
+3166337,Can I ask for a sliding scale based on my income when making an appointment with a psychologist?
+7009409,Where can I find directories of counsellors?
+7009409,Do I need a doctor's referral to see a counsellor?
+7009409,How can I find out about lower-cost options to access counselling in my community?
+7009409,Are counselling services at public mental health centers covered by MSP?
+7009409,How can Indigenous community members access counselling services locally or through the First Nations Health Authority?
+9630578,Where can I look for support groups specific to mental illnesses like mood disorders or post-traumatic stress disorder?
+9630578,How can I find out about support groups offered by the Canadian Mental Health Association or the BC Schizophrenia Society?
+9630578,Where can I find online support if I have trouble getting to an in-person support group?
+1585622,Where can I encourage my child to talk about their mental health concerns?
+1585622,Who can I talk to if I'm concerned about my child's mental health?
+1585622,Where can I find specific child and youth mental health services and providers?
+1585622,"Where can I get guidance, information, and support for youth and parent mental health?"
+1585622,Where can young people find easy-to-access mental health care and support?
+1833460,Where can I find information about local mental health services for older adults?
+1833460,Can I get information about senior's mental health resources from the Canadian Coalition for Seniors' Mental Health?
+1833460,Who can I contact if I'm concerned about a loved one's mental health?
+6981545,Where can I find self-help resources for depression that are appropriate for teens?
+6981545,Can you recommend some self-help materials for managing depression while pregnant or after delivery?
+6981545,Are there any online self-help programs for depression that are based outside of BC but offer good information?
+2115228,Where can I find self-help resources for anxiety if I'm not in British Columbia?
+2115228,Can I use Anxiety Canada's resources if I'm an adult but not a teen or young adult?
+2115228,What resources does the BC Reproductive Mental Health Program offer for anxiety during and after birth?
+2009922,Where can I find free programs to help manage ongoing physical or mental health issues?
+2009922,Are there any self-guided workbooks that discuss the links between physical health and mental health?
+2009922,"What resources are available to help me cope with depression, low mood, or anxiety caused by a chronic health condition?"
+2009922,Can I purchase any booklets that offer practical tips to cope with a chronic health problem?
+2009922,Where can I find one-on-one support from a therapist or counselor if I'm having a hard time coping?
+6167248,How can I determine if my substance use is causing problems?
+6167248,Where can I find resources for helping someone else with their substance use?
+6167248,What are some ways I can think about my relationship with substances?
+6167248,What tips and resources are available for cutting back or quitting substance use?
+6167248,How can I contact the Alcohol and Drug Information Referral Service for help in my area?
+6779222,How can I find out if my BC Medical Services Plan covers mental health services like counselling or therapy?
+6779222,Where can I find mental health clinics or hospitals that offer free or low-cost counselling services?
+6779222,How can I find mental health services in my area using HealthLink BC’s Find Services tool and Locator app?
+6779222,Where can I look for private mental health services that offer a sliding scale based on income?
+6779222,How can mental health organizations like Anxiety Canada or the Canadian Mental Health Association help me find local mental health services or connections?
+1546812,Is there a program to help me pay for my psychiatric medications?
+1546812,How can I apply for the Psychiatric Medications Plan through BC PharmaCare?
+1546812,How long does the coverage for Plan G typically last?
+2176317,Are there any mental health services covered by MSP that I could access for free?
+2176317,What are some ways I can offset the costs of mental health services not covered by MSP?
+2176317,How can I find mental health providers that offer lower-cost services or a sliding scale based on income?
+3284724,What are the different income levels for provincial income assistance?
+3284724,How can mental illness be considered in PWD or PPMB income assistance applications?
+3284724,Where can I find resources to help me navigate the application process for PWD and PPMB benefits?
+5981663,What is the purpose of MSP in British Columbia?
+5981663,How are MSP fees determined and paid?
+5981663,Where can I get help with applying for or managing my MSP coverage?
+6623577,What does it mean when someone recommends me to another service in healthcare?
+6623577,How do I access a specialist like a psychiatrist for mental health in BC?
+6623577,What is the process for getting an appointment with a specialist after a referral from my doctor?
+6851366,"Are psychiatrists able to prescribe medications, while psychologists cannot?"
+6851366,How are the education and training different between psychiatrists and registered psychologists?
+6851366,How does payment for services differ between accessing a psychologist in the public system versus the private system?
+1898078,What are the commonalities between psychotherapy and counseling?
+1898078,Can you explain the role of a professional therapist or counselor in the healing process?
+1898078,What are some examples of different psychotherapies that have been found effective for various problems or illnesses?
+1706961,What is the main focus of CBT in contrast to DBT?
+1706961,How does DBT deals with emotions and relationships differently than CBT?
+1706961,In what kind of mental health issues have DBT been proven to be effective?
+4925221,What are the different classes of antidepressant medications and how do they work?
+4925221,What are some common medications for each class of antidepressants?
+4925221,What factors might affect how a medication works for different people?
+1043721,How can I bring up my concerns about my diagnosis with the professional who provided it?
+1043721,What is the process for getting a second opinion from a different professional?
+1043721,How can I discuss my options and get a referral to a different program or health professional for a second opinion?
+8471978,What should I do if I don't think my prescribed antidepressant is working?
+8471978,Who can I contact if I want to discuss the common side effects of my medication?
+8471978,Is it safe to stop taking my psychiatric medication or change the dose without consulting a health care provider?
+8471978,What should I do if I experience a serious allergic reaction to my medication?
+8471978,How can I effectively communicate with my doctor about symptoms or side effects that I find embarrassing?
+2746837,How can I help an adult in my life who is struggling with potential mental health issues?
+2746837,"What are the risks of diagnosing a health condition, especially online, without professional input?"
+2746837,What is the role of the person being treated in their own recovery and how can I support them in this process?
+2746837,How can I effectively express my concerns and work together with the person to find a solution?
+2746837,What should I know about the BC Mental Health Act in relation to getting help for someone who is at risk of harming themselves or others?
+5640861,Where can I find a family doctor to talk about my concerns for my child or teenager?
+5640861,What kind of services and support can I expect from a school counselor regarding my child's mental health?
+5640861,What is the role of Kelty Mental Health Resource Centre and how can their Help Finder tool assist me in navigating the mental health system for my child?
+5640861,Does the Canadian Mental Health Association’s BC Division offer any resources or programs for managing behavior problems in children ages 3 to 12?
+5640861,Where can I find HeretoHelp resources on child and youth mental health and substance use?
+3393534,Where can I find more information about mental health problems that may arise from a brain injury?
+3393534,Are there service providers who are knowledgeable about both brain injuries and mental health?
+3393534,How can I find local and regional groups in BC that support those with brain injuries?
+3393534,What organizations provide information and support for those affected by brain injuries in BC?
+3393534,Are mental health challenges related to brain injuries always treatable?
+3839472,How can I recognize and empathize with my loved one's experience with depression?
+3839472,Where can I learn more about depression to better understand it?
+3839472,How can I create a safe space for my loved one to talk about their depression?
+3839472,"What are some things I should avoid, like giving unsolicited advice?"
+3839472,How can I support my loved one in their treatment and recovery?
+3055896,What are some trustworthy resources where I can learn more about schizophrenia?
+3055896,"How can I support my loved one during episodes of psychosis, especially when their hallucinations or delusions are distressing?"
+3055896,What is the role of a family member in helping a loved one manage their treatment for schizophrenia?
+3055896,What can I do if my loved one refuses treatment for schizophrenia and I believe they are at risk of harming themselves or others?
+3055896,What should I do if my loved one expresses thoughts of ending their life?
+4200873,How can learning about my parent's mental illness make it easier for me to understand their experiences?
+4200873,What self-care activities can I do to take care of myself while my parent is unwell?
+4200873,Where can I find programs specifically designed for young people in my situation?
+1337085,What resources are available for someone experiencing thoughts of suicide?
+1337085,How can I help someone I care about who is talking about suicide?
+1337085,What should I do in a situation where there is immediate danger or risk of harm due to suicide thoughts?
+2447683,What are the potential benefits and harms of using cannabis?
+2447683,How can the timing of when a person starts using cannabis impact potential harms?
+2447683,Are there certain groups of people who should avoid using cannabis due to increased risk of psychosis or schizophrenia?
+2447683,How can mixing drugs like cannabis and alcohol increase the possibility of experiencing harms?
+2447683,What is a simple way to gauge the use of psychoactive substances and prevent potential harms?
+7995219,Can you explain more about what mental well-being entails?
+7995219,How would you define a mental illness and how does it affect a person's life?
+7995219,Can a person have good mental health even if they have been diagnosed with a mental illness?
+7995219,"How does mental health exist on a continuum, similar to physical health?"
+7995219,What supports and tools can help someone live well and find meaning despite mental health challenges?
+2903196,Where can I find info sheets on mood disorders and bipolar disorder?
+2903196,Do you have a self-test for depression that also screens for signs of bipolar disorder?
+2903196,Can I find a series to help me deal with a diagnosis of depression and work with my doctor?
+2903196,Are there any books or resources with personal stories about depression?
+2903196,How can I access the What is Depression? booklet with audio in plain language for lower literacy readers?
+4893447,How can I distinguish between normal anxiety and an anxiety disorder?
+4893447,What are some examples of how anxiety disorders can impact someone's daily life?
+4893447,What are the differences between the former and current classifications of some anxiety disorders?
+8612349,What are the usual causes of stress?
+8612349,How is anxiety different from stress in terms of what it focuses on?
+8612349,Are the physical sensations of stress and anxiety always similar?
+8612349,"Can anxiety be triggered by a specific threat, or is it always about vague worries?"
+8612349,How can stress and anxiety impact well-being and daily life if they last for a long time?
+6062232,How is sadness different from the illness of depression?
+6062232,Can feelings of sadness impact daily life in a significant way like depression does?
+6062232,Are there any physical symptoms associated with depression that don't occur with normal sadness?
+9045344,How does persistent depressive disorder (PDD) differ from major depressive disorder in terms of symptoms and duration?
+9045344,What are the symptoms of PDD?
+9045344,Why is it a misconception that PDD is less severe than major depressive disorder?
+9045344,How can PDD affect a person's ability to work and enjoy hobbies?
+9045344,Why might someone with PDD think it is just part of their personality?
+9020440,What is the relationship between cyclothymic disorder and bipolar disorder?
+9020440,What are the symptoms of hypomania and depression associated with cyclothymic disorder?
+9020440,How does the frequency of symptoms in cyclothymic disorder compare to bipolar disorder?
+9020440,What wellness period are people with cyclothymic disorder likely to experience?
+9020440,Who should I consult if I suspect I have cyclothymic disorder?
+5819325,What does it mean when someone experiences four or more episodes of depression and/or mania in one year?
+5819325,Who can experience rapid cycling and when does it typically occur?
+5819325,Is rapid cycling a permanent condition for those diagnosed with bipolar disorder?
+4816493,Where can I find more information about the five groups of symptoms for borderline personality disorder?
+4816493,Do you know of any support groups or resources for borderline personality disorder in cities other than Vancouver and Victoria?
+4816493,Can you recommend any helpful books for managing borderline personality disorder?
+9178453,How does schizoid personality disorder affect social interactions and relationships?
+9178453,What is the difference between schizoid personality disorder and schizophrenia?
+9178453,How is schizoid personality disorder typically treated and managed?
+9679704,How does antisocial personality disorder affect a person's thoughts and behaviors?
+9679704,What is the relationship between conduct disorder and antisocial personality disorder?
+9679704,How can treatment help someone with antisocial personality disorder?
+9167296,"How does a personality disorder, such as obsessive-compulsive personality disorder, affect a person's daily life and thoughts?"
+9167296,What are the key differences between obsessive-compulsive personality disorder and obsessive-compulsive disorder?
+9167296,How is treatment typically approached for individuals with obsessive-compulsive personality disorder?
+9167296,What kind of distress might a person with obsessive-compulsive personality disorder experience when they can't achieve control or perfectionism?
+9167296,Where can someone seek help for obsessive-compulsive personality disorder in British Columbia?
+9049618,What are the main symptoms of binge-eating disorder?
+9049618,How can I recognize if someone has binge-eating disorder?
+9049618,What makes binge-eating disorder different from just overeating?
+9049618,How is binge-eating disorder treated?
+9049618,Who should I consult if I think I have binge-eating disorder?
+7984793,How does the 'split mind' in schizophrenia differ from the fragmented identity in dissociative identity disorder?
+7984793,What is the significance of dissociation in dissociative identity disorder?
+7984793,How does schizophrenia affect a person's perception of reality differently than dissociative identity disorder?
+7984793,"What are some common misconceptions about dissociative identity disorder, such as its portrayal in media, and how does it contrast with the actual experience of those diagnosed with it?"
+7984793,"How does schizophrenia impact an individual's thoughts, speech, and emotional expression as opposed to dissociative identity disorder?"
+6085633,What is the difference between psychosis and schizophrenia in terms of their definitions?
+6085633,"What are some common symptoms of psychosis, such as hallucinations and delusions?"
+6085633,"What mental illnesses or factors, aside from schizophrenia, can cause psychosis?"
+1896541,What do positive symptoms of schizophrenia involve?
+1896541,Can you explain what negative symptoms of schizophrenia are?
+1896541,What are cognitive symptoms and how do they relate to schizophrenia?
+4962901,Can you explain what a prodrome is in simpler terms?
+4962901,What are some early signs or symptoms of psychosis?
+4962901,Why is it important to seek help early if you or a loved one is experiencing symptoms of a mental health problem?
+6869967,How can I find a healthcare professional who specializes in assessing and treating ADHD in adults?
+6869967,What resources and support are available for adults with ADHD in Canada?
+6869967,Is there a support group for adults with ADHD in the Vancouver area?
+4824231,What does it mean when someone says they are addicted to something like shopping?
+4824231,How does the evidence suggest substance use and addiction differ?
+4824231,Can you explain how substance dependence develops and what it involves?
+4294616,What are some ways I can get help for an alcohol or drug use problem?
+4294616,How does my health and social situation factor into finding the right help for me?
+4294616,Can I talk to a nurse or doctor confidentially about drug use and what resources are available in my area?
+4373204,How can I determine if I'm drinking too much considering my unique relationship with alcohol?
+4373204,What are the signs of alcohol use becoming a problem in my life or in the lives of others?
+4373204,Where can I get personalized feedback about my drinking pattern as compared to Canada's Low-Risk Alcohol Drinking Guidelines?
+7807643,"If cannabis smoke contains harmful toxins, why is the risk of cancer less for cannabis smokers than tobacco smokers?"
+7807643,How do the negative effects of cannabis on the brain compare to those of alcohol?
+7807643,What are the benefits of legalizing cannabis in terms of minimizing potential harms?
+4352464,How can I help my child develop skills to make good choices regarding drugs?
+4352464,What are some ways to strengthen my relationship with my child through discussions about drugs?
+4352464,How can I support my child in developing critical thinking skills about drug use?
+6521784,What are the medical conditions that CBD has been used in the treatment of?
+6521784,How does CBD produce therapeutic effects in the body?
+6521784,Are there any recorded negative drug interactions with CBD?
+6521784,What are some potential side effects of using CBD?
+6521784,Is there any research on the use of CBD for Alzheimer's disease?
+3221856,What are the potential harms of vaping?
+3221856,How is nicotine delivered through vaping?
+3221856,What is the impact of vaping marketing strategies on young people?

docker-compose.yaml

@@ -0,0 +1,48 @@
+services:
+  postgres:
+    image: postgres:13
+    environment:
+      POSTGRES_DB: ${DB_NAME}
+      POSTGRES_USER: ${DB_USER}
+      POSTGRES_PASSWORD: ${DB_PASSWORD}
+    ports:
+      - "5432:5432"
+    volumes:
+      - pgdata:/var/lib/postgresql/data  # Using Docker's volume instead of bind mount
+
+  streamlit:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    ports:
+      - "8501:8501"
+    depends_on:
+      - postgres
+    environment:
+      - DB_NAME=${DB_NAME}
+      - DB_USER=${DB_USER}
+      - DB_PASSWORD=${DB_PASSWORD}
+      - DB_HOST=postgres
+      - DB_PORT=5432
+      - GROQ_API_KEY=${GROQ_API_KEY}
+      - MISTRAL_API_KEY=${MISTRAL_API_KEY}
+      - DATA_PATH=/app/dataset/data.csv
+    volumes:
+      - ./dataset:/app/dataset
+
+  grafana:
+    image: grafana/grafana:latest
+    ports:
+      - "3000:3000"
+    depends_on:
+      - postgres
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
+      - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource
+    volumes:
+      - grafana-storage:/var/lib/grafana
+
+volumes:
+  pgdata:  # This is now a Docker-managed volume
+  grafana-storage:
+    driver: local

grafana/dashboard.json

@@ -0,0 +1,952 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": 1,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "custom": {
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
+            "inspect": false
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 5,
+      "options": {
+        "cellHeight": "sm",
+        "footer": {
+          "countRows": false,
+          "fields": "",
+          "reducer": [
+            "sum"
+          ],
+          "show": false
+        },
+        "showHeader": true
+      },
+      "pluginVersion": "11.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n  timestamp AS time,\r\n  question,\r\n  answer,\r\n  relevance\r\nFROM conversations\r\nWHERE timestamp BETWEEN $__timeFrom() AND $__timeTo()\r\nORDER BY timestamp DESC\r\nLIMIT 5",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Recent Conversations",
+      "type": "table"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "fillOpacity": 80,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineWidth": 1,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 3,
+      "options": {
+        "barRadius": 0,
+        "barWidth": 0.97,
+        "fullHighlight": false,
+        "groupWidth": 0.7,
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "right",
+          "showLegend": true
+        },
+        "orientation": "horizontal",
+        "showValue": "never",
+        "stacking": "none",
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        },
+        "xTickLabelRotation": 0,
+        "xTickLabelSpacing": 0
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n  model_used,\r\n  COUNT(*) as count\r\nFROM conversations\r\nWHERE timestamp BETWEEN $__timeFrom() AND $__timeTo()\r\nGROUP BY model_used",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Model Usage Panel",
+      "type": "barchart"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [],
+          "thresholds": {
+            "mode": "percentage",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 70
+              },
+              {
+                "color": "red",
+                "value": 85
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 6,
+      "options": {
+        "minVizHeight": 75,
+        "minVizWidth": 75,
+        "orientation": "auto",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showThresholdLabels": false,
+        "showThresholdMarkers": true,
+        "sizing": "auto"
+      },
+      "pluginVersion": "11.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n  SUM(CASE WHEN feedback > 0 THEN 1 ELSE 0 END) as thumbs_up,\r\n  SUM(CASE WHEN feedback < 0 THEN 1 ELSE 0 END) as thumbs_down\r\nFROM feedback\r\nWHERE timestamp BETWEEN $__timeFrom() AND $__timeTo()",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Feedback Stats",
+      "type": "gauge"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            }
+          },
+          "mappings": []
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 2,
+      "options": {
+        "displayLabels": [
+          "percent",
+          "name"
+        ],
+        "legend": {
+          "calcs": [],
+          "displayMode": "hidden",
+          "placement": "right",
+          "showLegend": false,
+          "values": []
+        },
+        "pieType": "pie",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n  relevance,\r\n  COUNT(*) as count\r\nFROM conversations\r\nWHERE timestamp BETWEEN $__timeFrom() AND $__timeTo()\r\nGROUP BY relevance",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Relevance Distribution",
+      "type": "piechart"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 8,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n    date_trunc('hour', f.timestamp) AS time,\r\n    SUM(CASE WHEN f.feedback > 0 THEN 1 ELSE 0 END) AS positive_feedback,\r\n    SUM(CASE WHEN f.feedback < 0 THEN 1 ELSE 0 END) AS negative_feedback\r\nFROM feedback f\r\nWHERE f.timestamp >= $__timeFrom() AND f.timestamp < $__timeTo()\r\nGROUP BY 1\r\nORDER BY 1;\r\n",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Feedback Trends Over time",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 16
+      },
+      "id": 1,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n  timestamp AS time,\r\n  response_time\r\nFROM conversations\r\nWHERE timestamp BETWEEN $__timeFrom() AND $__timeTo()\r\nORDER BY timestamp",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Response Time",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 24
+      },
+      "id": 4,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n  $__timeGroup(timestamp, $__interval) AS time,\r\n  AVG(total_tokens) AS avg_tokens\r\nFROM conversations\r\nWHERE timestamp BETWEEN $__timeFrom() AND $__timeTo()\r\nGROUP BY 1\r\nORDER BY 1",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Token Usage Panel",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "custom": {
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
+            "inspect": false
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 24
+      },
+      "id": 9,
+      "options": {
+        "cellHeight": "sm",
+        "footer": {
+          "countRows": false,
+          "fields": "",
+          "reducer": [
+            "sum"
+          ],
+          "show": false
+        },
+        "showHeader": true
+      },
+      "pluginVersion": "11.2.0",
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n    question,\r\n    COUNT(*) AS frequency\r\nFROM conversations\r\nWHERE timestamp >= $__timeFrom() AND timestamp < $__timeTo()\r\nGROUP BY question\r\nORDER BY frequency DESC\r\nLIMIT 5;",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Most Frequent Questions",
+      "type": "table"
+    },
+    {
+      "datasource": {
+        "default": true,
+        "type": "grafana-postgresql-datasource",
+        "uid": "ae169klpan0g0e"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "barWidthFactor": 0.6,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 32
+      },
+      "id": 7,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "grafana-postgresql-datasource",
+            "uid": "ae169klpan0g0e"
+          },
+          "editorMode": "code",
+          "format": "table",
+          "rawQuery": true,
+          "rawSql": "SELECT\r\n    date_trunc('hour', timestamp) AS time,\r\n    COUNT(*) AS conversation_count\r\nFROM conversations\r\nWHERE timestamp >= $__timeFrom() AND timestamp < $__timeTo()\r\nGROUP BY 1\r\nORDER BY 1;",
+          "refId": "A",
+          "sql": {
+            "columns": [
+              {
+                "parameters": [],
+                "type": "function"
+              }
+            ],
+            "groupBy": [
+              {
+                "property": {
+                  "type": "string"
+                },
+                "type": "groupBy"
+              }
+            ],
+            "limit": 50
+          }
+        }
+      ],
+      "title": "Conversation count over time",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "",
+  "schemaVersion": 39,
+  "tags": [],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "Mental_Health",
+  "uid": "ce16auu7hu4n4f",
+  "version": 13,
+  "weekStart": ""
+}

grafana/init.py

@@ -0,0 +1,237 @@
+import os
+import time
+import requests
+import json
+import logging
+from typing import Dict, Any
+
+# Set up logging
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
+)
+logger = logging.getLogger(__name__)
+
+class GrafanaInitializer:
+    def __init__(self):
+        self.grafana_url = os.getenv('GRAFANA_URL', 'http://localhost:3000')
+        self.grafana_user = os.getenv('GRAFANA_ADMIN_USER', 'admin')
+        self.grafana_password = os.getenv('GRAFANA_ADMIN_PASSWORD', 'admin')
+        self.postgres_host = os.getenv('POSTGRES_HOST', 'postgres')
+        self.postgres_port = os.getenv('POSTGRES_PORT', '5432')
+        self.postgres_db = os.getenv('POSTGRES_DB', 'mental_health')
+        self.postgres_user = os.getenv('POSTGRES_USER', 'newton')
+        self.postgres_password = os.getenv('POSTGRES_PASSWORD', 'Admin')
+        
+        self.headers = {
+            'Content-Type': 'application/json',
+            'Accept': 'application/json',
+        }
+        self.session = requests.Session()
+        self.session.auth = (self.grafana_user, self.grafana_password)
+        
+    def wait_for_grafana(self, max_retries: int = 30, delay: int = 5) -> bool:
+        """Wait for Grafana to become available."""
+        for i in range(max_retries):
+            try:
+                response = self.session.get(f"{self.grafana_url}/api/health")
+                if response.status_code == 200:
+                    logger.info("Grafana is available")
+                    return True
+            except requests.exceptions.RequestException:
+                pass
+            logger.info(f"Waiting for Grafana to become available (attempt {i + 1}/{max_retries})")
+            time.sleep(delay)
+        return False
+
+    def setup_datasource(self) -> None:
+        """Configure PostgreSQL as a data source."""
+        datasource_config = {
+            "name": "PostgreSQL",
+            "type": "postgres",
+            "url": f"{self.postgres_host}:{self.postgres_port}",
+            "access": "proxy",
+            "basicAuth": False,
+            "database": self.postgres_db,
+            "user": self.postgres_user,
+            "secureJsonData": {
+                "password": self.postgres_password
+            },
+            "jsonData": {
+                "sslmode": "disable",
+                "maxOpenConns": 100,
+                "maxIdleConns": 100,
+                "connMaxLifetime": 14400,
+                "postgresVersion": 1300,
+                "timescaledb": False
+            }
+        }
+
+        response = self.session.post(
+            f"{self.grafana_url}/api/datasources",
+            json=datasource_config
+        )
+        
+        if response.status_code == 200:
+            logger.info("PostgreSQL datasource configured successfully")
+        else:
+            logger.error(f"Failed to configure datasource: {response.text}")
+
+    def create_dashboards(self) -> None:
+        """Initialize Grafana dashboards for system monitoring."""
+        # System Performance Dashboard
+        performance_dashboard = {
+            "dashboard": {
+                "title": "Mental Health Assistant Performance",
+                "timezone": "browser",
+                "panels": [
+                    # Response Time Panel
+                    {
+                        "title": "Average Response Time",
+                        "type": "graph",
+                        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
+                        "targets": [{
+                            "rawSql": """
+                            SELECT
+                                timestamp as time,
+                                avg(response_time) as "Response Time"
+                            FROM conversations
+                            WHERE $__timeFilter(timestamp)
+                            GROUP BY timestamp
+                            ORDER BY timestamp
+                            """,
+                            "format": "time_series"
+                        }]
+                    },
+                    # Model Usage Panel
+                    {
+                        "title": "Model Usage Distribution",
+                        "type": "piechart",
+                        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
+                        "targets": [{
+                            "rawSql": """
+                            SELECT
+                                model_used as metric,
+                                count(*) as value
+                            FROM conversations
+                            WHERE $__timeFilter(timestamp)
+                            GROUP BY model_used
+                            """
+                        }]
+                    },
+                    # Token Usage Panel
+                    {
+                        "title": "Average Token Usage",
+                        "type": "graph",
+                        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 8},
+                        "targets": [{
+                            "rawSql": """
+                            SELECT
+                                timestamp as time,
+                                avg(total_tokens) as "Total Tokens",
+                                avg(prompt_tokens) as "Prompt Tokens",
+                                avg(completion_tokens) as "Completion Tokens"
+                            FROM conversations
+                            WHERE $__timeFilter(timestamp)
+                            GROUP BY timestamp
+                            ORDER BY timestamp
+                            """,
+                            "format": "time_series"
+                        }]
+                    }
+                ],
+                "refresh": "5s"
+            }
+        }
+
+        # User Feedback Dashboard
+        feedback_dashboard = {
+            "dashboard": {
+                "title": "User Feedback Analysis",
+                "timezone": "browser",
+                "panels": [
+                    # Feedback Distribution
+                    {
+                        "title": "Feedback Distribution",
+                        "type": "stat",
+                        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
+                        "targets": [{
+                            "rawSql": """
+                            SELECT
+                                sum(case when feedback > 0 then 1 else 0 end) as "Positive",
+                                sum(case when feedback < 0 then 1 else 0 end) as "Negative"
+                            FROM feedback
+                            WHERE $__timeFilter(timestamp)
+                            """
+                        }]
+                    },
+                    # Relevance Distribution
+                    {
+                        "title": "Answer Relevance Distribution",
+                        "type": "piechart",
+                        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
+                        "targets": [{
+                            "rawSql": """
+                            SELECT
+                                relevance as metric,
+                                count(*) as value
+                            FROM conversations
+                            WHERE $__timeFilter(timestamp)
+                            GROUP BY relevance
+                            """
+                        }]
+                    },
+                    # Feedback Timeline
+                    {
+                        "title": "Feedback Timeline",
+                        "type": "graph",
+                        "gridPos": {"h": 8, "w": 24, "x": 0, "y": 8},
+                        "targets": [{
+                            "rawSql": """
+                            SELECT
+                                date_trunc('hour', timestamp) as time,
+                                count(*) FILTER (WHERE feedback > 0) as "Positive",
+                                count(*) FILTER (WHERE feedback < 0) as "Negative"
+                            FROM feedback
+                            WHERE $__timeFilter(timestamp)
+                            GROUP BY date_trunc('hour', timestamp)
+                            ORDER BY time
+                            """,
+                            "format": "time_series"
+                        }]
+                    }
+                ],
+                "refresh": "5s"
+            }
+        }
+
+        # Create dashboards
+        for dashboard in [performance_dashboard, feedback_dashboard]:
+            response = self.session.post(
+                f"{self.grafana_url}/api/dashboards/db",
+                json=dashboard
+            )
+            
+            if response.status_code == 200:
+                logger.info(f"Dashboard '{dashboard['dashboard']['title']}' created successfully")
+            else:
+                logger.error(f"Failed to create dashboard: {response.text}")
+
+def main():
+    initializer = GrafanaInitializer()
+    
+    # Wait for Grafana to become available
+    if not initializer.wait_for_grafana():
+        logger.error("Grafana is not available after maximum retries")
+        return
+    
+    # Set up data source and dashboards
+    try:
+        initializer.setup_datasource()
+        initializer.create_dashboards()
+        logger.info("Grafana initialization completed successfully")
+    except Exception as e:
+        logger.error(f"Failed to initialize Grafana: {str(e)}")
+
+if __name__ == "__main__":
+    main()

notebooks/.ipynb_checkpoints/data_generation-checkpoint.ipynb

@@ -0,0 +1,542 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "ae13d278-e1c5-496e-8871-3936ccb71b5d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from mistralai import Mistral\n",
+    "import pandas as pd\n",
+    "import minsearch\n",
+    "import os\n",
+    "from groq import Groq\n",
+    "from dotenv import load_dotenv\n",
+    "from tqdm.auto import tqdm\n",
+    "import pandas as pd\n",
+    "import minsearch\n",
+    "import os\n",
+    "from groq import Groq\n",
+    "from dotenv import load_dotenv\n",
+    "from tqdm.auto import tqdm\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "048c2b93-82c0-4d5b-bcaa-47b8531506cf",
+   "metadata": {},
+   "source": [
+    "## Environment variables Loading"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "7c5f1265-31da-4ad6-bd8c-bd912b39e5b1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "True"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "load_dotenv()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "4c4fa421-e6b2-48a3-9329-215c91c26ade",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Access the API keys\n",
+    "mistral_api_key = os.getenv(\"MISTRAL_API_KEY\")\n",
+    "groq_api_key = os.getenv(\"GROQ_API_KEY\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "722521c0-1558-430a-852f-a5afdd082a49",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Variable          Type        Data/Info\n",
+      "---------------------------------------\n",
+      "Groq              type        <class 'groq.Groq'>\n",
+      "Mistral           type        <class 'mistralai.sdk.Mistral'>\n",
+      "groq_api_key      str         gsk_32VwUDXWlbcss91AZEhIW<...>Yv5D56tMak8RwoyfSBXePPdht\n",
+      "load_dotenv       function    <function load_dotenv at 0x0000024D110E7430>\n",
+      "minsearch         module      <module 'minsearch' from <...>notebooks\\\\minsearch.py'>\n",
+      "mistral_api_key   str         8liS0F7zb7zNgujwgo3dgTbr6HbfvmeI\n",
+      "os                module      <module 'os' from 'C:\\\\Us<...>n\\\\Python39\\\\lib\\\\os.py'>\n",
+      "pd                module      <module 'pandas' from 'C:<...>es\\\\pandas\\\\__init__.py'>\n",
+      "tqdm              type        <class 'tqdm.auto.tqdm'>\n"
+     ]
+    }
+   ],
+   "source": [
+    "%whos"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "069f3cf6-4941-46f5-adeb-c56813c8db3f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df=pd.read_csv(\"../dataset/Mental_Health_FAQ.csv\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "a8e4b926-85a9-4130-8107-d416abd56a97",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Question_ID</th>\n",
+       "      <th>Questions</th>\n",
+       "      <th>Answers</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>1590140</td>\n",
+       "      <td>What does it mean to have a mental illness?</td>\n",
+       "      <td>Mental illnesses are health conditions that di...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>2110618</td>\n",
+       "      <td>Who does mental illness affect?</td>\n",
+       "      <td>It is estimated that mental illness affects 1 ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>6361820</td>\n",
+       "      <td>What causes mental illness?</td>\n",
+       "      <td>It is estimated that mental illness affects 1 ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>9434130</td>\n",
+       "      <td>What are some of the warning signs of mental i...</td>\n",
+       "      <td>Symptoms of mental health disorders vary depen...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>7657263</td>\n",
+       "      <td>Can people with mental illness recover?</td>\n",
+       "      <td>When healing from mental illness, early identi...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "   Question_ID                                          Questions  \\\n",
+       "0      1590140        What does it mean to have a mental illness?   \n",
+       "1      2110618                    Who does mental illness affect?   \n",
+       "2      6361820                        What causes mental illness?   \n",
+       "3      9434130  What are some of the warning signs of mental i...   \n",
+       "4      7657263            Can people with mental illness recover?   \n",
+       "\n",
+       "                                             Answers  \n",
+       "0  Mental illnesses are health conditions that di...  \n",
+       "1  It is estimated that mental illness affects 1 ...  \n",
+       "2  It is estimated that mental illness affects 1 ...  \n",
+       "3  Symptoms of mental health disorders vary depen...  \n",
+       "4  When healing from mental illness, early identi...  "
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "5c69465d-0018-432f-b134-2fa929045e83",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "documents=df.to_dict(orient='records')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "5297116d-cf4b-44ae-aaad-72079a906855",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'Question_ID': 6361820,\n",
+       " 'Questions': 'What causes mental illness?',\n",
+       " 'Answers': 'It is estimated that mental illness affects 1 in 5 adults in America, and that 1 in 24 adults have a serious mental illness. Mental illness does not discriminate; it can affect anyone, regardless of gender, age, income, social status, ethnicity, religion, sexual orientation, or background. Although mental illness can affect anyone, certain conditions may be more common in different populations. For instance, eating disorders tend to occur more often in females, while disorders such as attention deficit/hyperactivity disorder is more prevalent in children. Additionally, all ages are susceptible, but the young and the old are especially vulnerable. Mental illnesses usually strike individuals in the prime of their lives, with 75 percent of mental health conditions developing by the age of 24. This makes identification and treatment of mental disorders particularly difficult, because the normal personality and behavioral changes of adolescence may mask symptoms of a mental health condition. Parents and caretakers should be aware of this fact, and take notice of changes in their child’s mood, personality, personal habits, and social withdrawal. When these occur in children under 18, they are referred to as serious emotional disturbances (SEDs).'}"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "documents[2]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "604c32fc-bf9b-465d-b2d6-e55e8947ef4d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompt_template=\"\"\"You are emulating a patient concerned about mental health. For each answer, \n",
+    "formulate 5 questions that this patient might ask. The questions should be complete, not too short, \n",
+    "and use as few words as possible from the original answer.\n",
+    "\n",
+    "The record:\n",
+    "Question_ID:{Question_ID}\n",
+    "Questions:{Questions}\n",
+    "Answer: {Answers}\n",
+    "\n",
+    "Provide the output in parsable JSON format without using code blocks. The output should be structured like this:\n",
+    "\"\"\".strip()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "dacc3ea2-df3b-4fad-b9cd-1cc3a88a6e1c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompt=prompt_template.format(**documents[0])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "53b634ad-dc6b-4848-b6b4-b882e90076cb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = Mistral(api_key=mistral_api_key)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "079bc5f5-2977-43da-8eca-c340dcb884d4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def llm(prompt,model=\"mistral-large-latest\"):\n",
+    "        response = client.chat.complete(\n",
+    "        model=model,\n",
+    "        messages=[{\"role\": \"user\", \"content\": prompt}])\n",
+    "        return response.choices[0].message.content\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "d227d497-bc50-49d8-a141-8e8a182aecf4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "quiz=llm(prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "e7a63537-5edd-475b-a032-e5bb7380e160",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "```json\n",
+      "{\n",
+      "  \"Question1\": \"What are the main aspects of life affected by mental illnesses?\",\n",
+      "  \"Question2\": \"How do mental illnesses vary in terms of severity and impact?\",\n",
+      "  \"Question3\": \"Are mental illnesses related to a person's character or intelligence?\",\n",
+      "  \"Question4\": \"How is mental illness treated, and what is the success rate?\",\n",
+      "  \"Question5\": \"Can someone with a mental illness lead an independent and successful life?\"\n",
+      "}\n",
+      "```\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(quiz)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "id": "a5f31769-9b1d-4ca2-a565-24298207bdb4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "  \"question1\": \"What is the definition of mental illness?\",\n",
+      "  \"question2\": \"How do mental illnesses affect a person's daily functioning?\",\n",
+      "  \"question3\": \"What are the different types of mental illnesses?\",\n",
+      "  \"question4\": \"What are the treatment options for mental illnesses?\",\n",
+      "  \"question5\": \"What is the impact of proper treatment on a person's life with mental illness?\"\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(quiz)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "id": "a2286fdc-116b-44c5-af27-21fdab4e8099",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n",
+      "3\n"
+     ]
+    }
+   ],
+   "source": [
+    "for quiz in documents:\n",
+    "    print(len(quiz))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "71dedb46-3b27-4aa0-a972-0907fae7a5b1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "377"
+      ]
+     },
+     "execution_count": 21,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(quiz)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5fbc9707-e590-4c86-bcfd-4374e07b659f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.info()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b611ea3d-ff42-49b0-82c3-c1c970cedcf2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from mistralai import Mistral\n",
+    "\n",
+    "client = Mistral(api_key=mistral_api_key)\n",
+    "\n",
+    "model = \"codestral-latest\"\n",
+    "message = [{\"role\": \"user\", \"content\": \"Write a function for fibonacci\"}]\n",
+    "chat_response = client.chat.complete(\n",
+    "    model = model,\n",
+    "    messages = message\n",
+    ")\n",
+    "print(chat_response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5b254557-6c35-4306-aada-bae7a264ba89",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/data_generation.ipynb

@@ -0,0 +1,1336 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ae13d278-e1c5-496e-8871-3936ccb71b5d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from mistralai import Mistral\n",
+    "import pandas as pd\n",
+    "import minsearch\n",
+    "import os\n",
+    "from groq import Groq\n",
+    "from dotenv import load_dotenv\n",
+    "from tqdm.auto import tqdm\n",
+    "import pandas as pd\n",
+    "import minsearch\n",
+    "import os\n",
+    "from groq import Groq\n",
+    "from dotenv import load_dotenv\n",
+    "from tqdm.auto import tqdm\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "048c2b93-82c0-4d5b-bcaa-47b8531506cf",
+   "metadata": {},
+   "source": [
+    "## Environment variables Loading"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "7c5f1265-31da-4ad6-bd8c-bd912b39e5b1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "True"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "load_dotenv()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "4c4fa421-e6b2-48a3-9329-215c91c26ade",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Access the API keys\n",
+    "mistral_api_key = os.getenv(\"MISTRAL_API_KEY\")\n",
+    "groq_api_key = os.getenv(\"GROQ_API_KEY\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "722521c0-1558-430a-852f-a5afdd082a49",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Variable          Type        Data/Info\n",
+      "---------------------------------------\n",
+      "Groq              type        <class 'groq.Groq'>\n",
+      "Mistral           type        <class 'mistralai.sdk.Mistral'>\n",
+      "groq_api_key      str         gsk_32VwUDXWlbcss91AZEhIW<...>Yv5D56tMak8RwoyfSBXePPdht\n",
+      "load_dotenv       function    <function load_dotenv at 0x00000145257C5310>\n",
+      "minsearch         module      <module 'minsearch' from <...>notebooks\\\\minsearch.py'>\n",
+      "mistral_api_key   str         8liS0F7zb7zNgujwgo3dgTbr6HbfvmeI\n",
+      "os                module      <module 'os' from 'C:\\\\Us<...>n\\\\Python39\\\\lib\\\\os.py'>\n",
+      "pd                module      <module 'pandas' from 'C:<...>es\\\\pandas\\\\__init__.py'>\n",
+      "tqdm              type        <class 'tqdm.auto.tqdm'>\n"
+     ]
+    }
+   ],
+   "source": [
+    "%whos"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "069f3cf6-4941-46f5-adeb-c56813c8db3f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df=pd.read_csv(\"../dataset/Mental_Health_FAQ.csv\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "a8e4b926-85a9-4130-8107-d416abd56a97",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Question_ID</th>\n",
+       "      <th>Questions</th>\n",
+       "      <th>Answers</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>1590140</td>\n",
+       "      <td>What does it mean to have a mental illness?</td>\n",
+       "      <td>Mental illnesses are health conditions that di...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>2110618</td>\n",
+       "      <td>Who does mental illness affect?</td>\n",
+       "      <td>It is estimated that mental illness affects 1 ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>6361820</td>\n",
+       "      <td>What causes mental illness?</td>\n",
+       "      <td>It is estimated that mental illness affects 1 ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>9434130</td>\n",
+       "      <td>What are some of the warning signs of mental i...</td>\n",
+       "      <td>Symptoms of mental health disorders vary depen...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>7657263</td>\n",
+       "      <td>Can people with mental illness recover?</td>\n",
+       "      <td>When healing from mental illness, early identi...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "   Question_ID                                          Questions  \\\n",
+       "0      1590140        What does it mean to have a mental illness?   \n",
+       "1      2110618                    Who does mental illness affect?   \n",
+       "2      6361820                        What causes mental illness?   \n",
+       "3      9434130  What are some of the warning signs of mental i...   \n",
+       "4      7657263            Can people with mental illness recover?   \n",
+       "\n",
+       "                                             Answers  \n",
+       "0  Mental illnesses are health conditions that di...  \n",
+       "1  It is estimated that mental illness affects 1 ...  \n",
+       "2  It is estimated that mental illness affects 1 ...  \n",
+       "3  Symptoms of mental health disorders vary depen...  \n",
+       "4  When healing from mental illness, early identi...  "
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "5c69465d-0018-432f-b134-2fa929045e83",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "documents=df.to_dict(orient='records')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "5297116d-cf4b-44ae-aaad-72079a906855",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'Question_ID': 1590140,\n",
+       " 'Questions': 'What does it mean to have a mental illness?',\n",
+       " 'Answers': 'Mental illnesses are health conditions that disrupt a person’s thoughts, emotions, relationships, and daily functioning. They are associated with distress and diminished capacity to engage in the ordinary activities of daily life.\\nMental illnesses fall along a continuum of severity: some are fairly mild and only interfere with some aspects of life, such as certain phobias. On the other end of the spectrum lie serious mental illnesses, which result in major functional impairment and interference with daily life. These include such disorders as major depression, schizophrenia, and bipolar disorder, and may require that the person receives care in a hospital.\\nIt is important to know that mental illnesses are medical conditions that have nothing to do with a person’s character, intelligence, or willpower. Just as diabetes is a disorder of the pancreas, mental illness is a medical condition due to the brain’s biology.\\nSimilarly to how one would treat diabetes with medication and insulin, mental illness is treatable with a combination of medication and social support. These treatments are highly effective, with 70-90 percent of individuals receiving treatment experiencing a reduction in symptoms and an improved quality of life. With the proper treatment, it is very possible for a person with mental illness to be independent and successful.'}"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "documents[0]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "604c32fc-bf9b-465d-b2d6-e55e8947ef4d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompt_template=\"\"\"You are emulating a patient who is concerned about mental health.\n",
+    "Based on the answer provided, formulate 3 questions that this patient might ask. \n",
+    "The questions should be complete, not too short, and use as few words as possible from the original answer.\n",
+    "\n",
+    "The record:\n",
+    "Question_ID:{Question_ID}\n",
+    "Questions: {Questions}\n",
+    "Answer: {Answers}\n",
+    "\n",
+    "Provide the output in parsable JSON format without using code blocks:\n",
+    "\n",
+    "{{[\"question1\", \"question2\",..., \"question5\"]}}\n",
+    "\n",
+    "\"\"\".strip()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 80,
+   "id": "dacc3ea2-df3b-4fad-b9cd-1cc3a88a6e1c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompt=prompt_template.format(**documents[0])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 120,
+   "id": "53b634ad-dc6b-4848-b6b4-b882e90076cb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = Groq(api_key=groq_api_key)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 81,
+   "id": "079bc5f5-2977-43da-8eca-c340dcb884d4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def llm(prompt,model=\"mixtral-8x7b-32768\"):\n",
+    "        response = client.chat.completions.create(\n",
+    "        model=model,\n",
+    "        messages=[{\"role\": \"user\", \"content\": prompt}])\n",
+    "        return response.choices[0].message.content\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 61,
+   "id": "d227d497-bc50-49d8-a141-8e8a182aecf4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "quiz=llm(prompt)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 62,
+   "id": "e7a63537-5edd-475b-a032-e5bb7380e160",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[\n",
+      "\"How do mental illnesses affect a person's daily life?\",\n",
+      "\"What are some examples of serious mental illnesses?\",\n",
+      "\"What should I understand about the cause of mental illness?\",\n",
+      "\"What are the common treatments for mental illness?\",\n",
+      "\"How effective are the treatments for mental illness and what is the quality of life people can expect after receiving treatment?\"\n",
+      "]\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(quiz)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "71dedb46-3b27-4aa0-a972-0907fae7a5b1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "666"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(quiz)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 121,
+   "id": "70bb6f1d-1bb8-49e8-8281-8476dc5d94f1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "prompt_template=\"\"\"You are emulating a patient who is concerned about mental health.\n",
+    "Based on the answer provided, formulate 3 questions that this patient might ask. \n",
+    "The questions should be complete, not too short, and use as few words as possible from the original answer.\n",
+    "\n",
+    "The record:\n",
+    "Question_ID:{Question_ID}\n",
+    "Questions: {Questions}\n",
+    "Answer: {Answers}\n",
+    "\n",
+    "Provide the output in parsable JSON without using code blocks:\n",
+    "\n",
+    "{{\"questions\": [\"question1\", \"question2\", ..., \"question5\"]}}\n",
+    "\n",
+    "\"\"\".strip()\n",
+    "def generate_data(doc):\n",
+    "    prompt=prompt_template.format(**doc)\n",
+    "    response = client.chat.completions.create(\n",
+    "        model='mixtral-8x7b-32768',\n",
+    "        messages=[{\"role\": \"user\", \"content\": prompt}])\n",
+    "    return response.choices[0].message.content\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 122,
+   "id": "338aa361-0f1d-4529-8598-10e5bab0f8b7",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "5e864b5bc273436caba87010785a2998",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "  0%|          | 0/98 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Raw JSON data for doc_id 1590140: {\n",
+      "\"questions\": [\n",
+      "\"How do mental illnesses affect a person's daily functioning and relationships?\",\n",
+      "\"What are some examples of serious mental illnesses that can result in major functional impairment?\",\n",
+      "\"Why is it a misconception to associate mental illness with a person's character, intelligence, or willpower?\",\n",
+      "\"How are mental illnesses treated, and what is the success rate of these treatments?\",\n",
+      "\"Can a person with mental illness become independent and successful with proper treatment?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2110618: {\n",
+      "\"questions\": [\n",
+      "\"Who are the groups most commonly affected by different mental illnesses?\",\n",
+      "\"How do mental illnesses affect children and the elderly?\",\n",
+      "\"What behaviors should parents and caretakers watch out for in children that may indicate a serious emotional disturbance?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6361820: {\n",
+      "\"questions\": [\n",
+      "\"How does mental illness affect different populations?\",\n",
+      "\"At what age are individuals most susceptible to mental illness?\",\n",
+      "\"What signs should parents and caretakers look out for in children under 18 that may indicate a mental health condition?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9434130: {\n",
+      "\"questions\": [\n",
+      "\"What are some general symptoms that may indicate a mental health disorder in adults?\",\n",
+      "\"Can you name some symptoms that could suggest a mental health issue in older children and pre-teens?\",\n",
+      "\"Are there any signs that might point to mental health problems in younger children?\",\n",
+      "\"What kind of behavior in children might suggest they are struggling with mental health?\",\n",
+      "\"What are some potential indicators of mental health issues related to mood or eating habits?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7657263: {\n",
+      "\"questions\": [\n",
+      "\"Can early identification and treatment improve recovery from mental illness?\",\n",
+      "\"Are there effective treatments available for different types of mental illnesses?\",\n",
+      "\"What is the importance of a person's active engagement in their own recovery process from mental illness?\",\n",
+      "\"Is it common for some people with mental illness to experience a return of symptoms after treatment?\",\n",
+      "\"How can one live a fulfilled and productive life despite careful monitoring and management of mental illness?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1619387: {\n",
+      "\"questions\": [\n",
+      "\"What should I do if I suspect someone I know has a mental disorder?\",\n",
+      "\"How can I help a friend or family member who might be experiencing symptoms of a mental health condition?\",\n",
+      "\"What are the benefits of speaking with a mental health professional if I think a loved one has symptoms of a mental disorder?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1030153: {\n",
+      "\"questions\": [\n",
+      "\"How can I search for mental health providers in my area?\",\n",
+      "\"What should I consider when researching mental health professionals for myself or my child?\",\n",
+      "\"How can I ensure that I will feel comfortable with the mental health professional I choose?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 8022026: {\n",
+      "\"questions\": [\n",
+      "\"What types of treatments are available for mental illness?\",\n",
+      "\"How do different treatment options work for different people?\",\n",
+      "\"How can I find the most effective treatment for myself or my child?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1155199: {\n",
+      "\"questions\": [\n",
+      "\"What can I expect in terms of the cause of my illness?\",\n",
+      "\"Are there other conditions that share my symptoms?\",\n",
+      "\"What kind of physical or neurological tests are usually involved?\",\n",
+      "\"What other specialists might be involved in my treatment and who will coordinate?\",\n",
+      "\"What is the typical role of the family in the treatment process?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7760466: {\n",
+      "\"questions\": [\n",
+      "\"What are the differences in education and training among mental health professionals?\",\n",
+      "\"How do mental health professionals specialize in different areas?\",\n",
+      "\"How can I find a mental health professional that fits my specific needs?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2553795: {\n",
+      "\"questions\": [\n",
+      "\"What steps can I take to find a mental health professional who is a good fit for me or my child?\",\n",
+      "\"How can I ensure that my child or I will feel comfortable working with a mental health professional?\",\n",
+      "\"What resources are available to help me research and find the right mental health professional for my needs or my child's needs?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1259439: {\n",
+      "\"questions\": [\n",
+      "\"What can you tell me about the cause of my illness?\",\n",
+      "\"Are there other conditions that share my symptoms?\",\n",
+      "\"What kinds of examinations do you usually conduct?\",\n",
+      "\"What is your recommended treatment program for my diagnosis?\",\n",
+      "\"What can I expect from the medications prescribed for my condition?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4197817: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find mental health resources in my area?\",\n",
+      "\"What kind of organizations offer support groups for consumers and their families?\",\n",
+      "\"Are support groups always led by mental health professionals or can they be peer led?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3388962: {\n",
+      "\"questions\": [\n",
+      "\"What is the primary purpose of the new medication?\",\n",
+      "\"How long does it typically take for the medication to take effect and how will I know when it is effective?\",\n",
+      "\"What are the proper directions for taking this medication, including any food, drinks, or activities to avoid?\",\n",
+      "\"What side effects should I be aware of and what should I do if they occur?\",\n",
+      "\"What is the procedure if I accidentally miss a dose of the medication?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 5343181: {\n",
+      "\"questions\": [\n",
+      "\"If I feel better after taking medication, does that mean I can stop taking it without consulting my doctor?\",\n",
+      "\"What might happen if I stop taking my medication abruptly?\",\n",
+      "\"How can I address side effects that make me unwilling to continue treatment?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 5778437: {\n",
+      "\"questions\": [\n",
+      "\"What organizations provide prescription assistance programs for individuals with financial needs?\",\n",
+      "\"Do I need a doctor's consent to apply for prescription assistance programs?\",\n",
+      "\"What kind of proof do I need to provide for these prescription assistance programs?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9541219: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find a therapist who specializes in my specific mental health condition?\",\n",
+      "\"Are there different types of therapy for children and adults?\",\n",
+      "\"What kind of therapy would be most effective for my symptoms?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3268009: {\n",
+      "\"questions\": [\n",
+      "\"What are some alternative types of mental health treatment?\",\n",
+      "\"Are there any self-help plans for mental health conditions?\",\n",
+      "\"What is the role of peer support in mental health treatment?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3340726: {\n",
+      "\"questions\": [\n",
+      "\"What are the different kinds of mental health professionals that I can choose from?\",\n",
+      "\"How can I find the right mental health professional for my needs?\",\n",
+      "\"Can you give me more information about the various types of mental health professionals?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9539480: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find peer support organizations?\",\n",
+      "\"Are there support groups led by mental health professionals?\",\n",
+      "\"How can I find support groups for my family and friends?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4211025: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find help for a mental health crisis?\",\n",
+      "\"What kinds of services does inpatient care provide for people in crisis?\",\n",
+      "\"How can inpatient care assist with adjusting to new medications or symptoms?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4031860: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find resources to help me locate mental health therapists in my area?\",\n",
+      "\"How can I learn about different mental health treatment options in my community?\",\n",
+      "\"Can you suggest any ways I can better understand the treatment process for mental health services?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 5215843: {\n",
+      "\"questions\": [\n",
+      "\"What are the risks associated with clinical trials?\",\n",
+      "\"How can I find out about new treatment approaches being tested?\",\n",
+      "\"Where can I get more information about participating in research studies?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 8927672: {\n",
+      "\"questions\": [\n",
+      "\"What is a psychiatric advance directive and how does it differ from a medical advance directive?\",\n",
+      "\"How can I create a psychiatric advance directive and who will it impact during a mental health crisis?\",\n",
+      "\"What kind of treatment instructions can I include in a psychiatric advance directive?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7728364: {\n",
+      "\"questions\": [\n",
+      "\"What exactly makes up mental health?\",\n",
+      "\"Can you explain the components of mental health?\",\n",
+      "\"How are beliefs, thoughts, feelings, and behaviors related to mental health?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4194958: {\n",
+      "\"questions\": [\n",
+      "\"What can I do if the usual support methods don't work for me?\",\n",
+      "\"How can I stay hopeful while searching for the right help?\",\n",
+      "\"What are some other approaches I can try if the current ones don't help?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1667863: {\n",
+      "\"questions\": [\n",
+      "\"Is it possible to take steps to avoid mental health issues?\",\n",
+      "\"How does working on our wellbeing and resilience contribute to preventing mental health problems?\",\n",
+      "\"What role does seeking help early play in preventing mental health issues from becoming severe?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 8904276: {\n",
+      "\"questions\": [\n",
+      "\"Are there any treatments that can help with mental health issues?\",\n",
+      "\"Is it possible to find relief from mental health problems through talking and counselling?\",\n",
+      "\"Can medication, friendships, exercise, sleep, and nutrition play a role in addressing mental health concerns?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4283807: {\n",
+      "\"questions\": [\"What are some psychological causes of mental health problems?\", \"What role does biology play in mental health issues?\", \"How can social factors contribute to mental health problems?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 8690253: {\n",
+      "\"questions\": [\n",
+      "\"Who are some people I can talk to if I'm worried about my mental health?\",\n",
+      "\"What steps can I take to find out more information about what I'm experiencing with my mental health?\",\n",
+      "\"How can talking to someone about my mental health help me get perspective and start getting help?\",\n",
+      "\"What kind of help can I expect to receive after I talk to someone about my mental health?\",\n",
+      "\"What can I do to get started on the path of improving my mental health?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2973656: {\n",
+      "\"questions\": [\n",
+      "\"How can I tell if my beliefs and thoughts are affecting my daily functioning?\",\n",
+      "\"What are the signs that my feelings and behaviors are having a significant impact on my ability to function normally?\",\n",
+      "\"When is it important to seek help if my mental health is affecting my daily life?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4759773: {\n",
+      "\"questions\": [\n",
+      "\"What actions can I take if I'm concerned about a loved one's mental health?\",\n",
+      "\"How can I approach a friend or relative about seeking mental health support?\",\n",
+      "\"What is the first step I should take when I notice a concern for someone's mental well-being?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1511075: {\n",
+      "\"questions\": [\n",
+      "\"How should I handle people giving me advice based on evidence?\",\n",
+      "\"What is the best way to find out what works for me when people tell me what to do?\",\n",
+      "\"How can I balance following others' advice and doing what is best for my mental health?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7069853: {\n",
+      "\"questions\": [\n",
+      "\"Why is it important for my mental health to stay informed about the pandemic?\",\n",
+      "\"How can reliable and up-to-date information about the coronavirus help me feel more resilient?\",\n",
+      "\"What are the negative effects of following the news all day and how can I avoid them?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4181750: {\n",
+      "\"questions\": [\n",
+      "\"How can I determine if a source is reliable for information about the coronavirus and COVID-19?\",\n",
+      "\"What should I consider when evaluating the credibility of experts quoted or cited in health articles?\",\n",
+      "\"How can I identify if a source is trying to make me feel a certain way rather than providing factual information?\",\n",
+      "\"What is the role of social media in obtaining reliable health information and what cautions should I take?\",\n",
+      "\"Where can I learn more about digital and media literacy and thinking critically around health claims?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2554141: {\n",
+      "\"questions\": [\n",
+      "\"Why does having a plan help manage fear and stress during a pandemic?\",\n",
+      "\"What are some tasks I can include in my daily schedule for managing during a pandemic?\",\n",
+      "\"Where can I find resources on problem-solving to help me manage during a pandemic?\",\n",
+      "\"How can I practice problem-solving to manage uncertain situations during a pandemic?\",\n",
+      "\"What are some ways I can manage feelings of being overwhelmed or hopeless during a pandemic?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9100298: {\n",
+      "\"questions\": [\n",
+      "\"What are some ways I can maintain social connections while physically separated from loved ones?\",\n",
+      "\"What should I do if I'm feeling lonely during this time?\",\n",
+      "\"Where can I find local community support groups and mutual aid groups online?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2884887: {\n",
+      "\"questions\": [\n",
+      "\"How does taking care of my physical health impact my mental health?\",\n",
+      "\"What are some tips for maintaining a balanced diet while spending more time at home?\",\n",
+      "\"What are some ways to stay active and exercise while staying safe at home?\",\n",
+      "\"How can I practice healthy sleep habits if stress and anxiety are affecting my sleep?\",\n",
+      "\"What should I do if I have a chronic illness and need to take additional precautions during this time?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9250044: {\n",
+      "\"questions\": [\n",
+      "\"What are some low-effort distractions I can try when I'm feeling overwhelmed or isolated?\",\n",
+      "\"What are some ways I can keep myself engaged and focused if I have more energy?\",\n",
+      "\"How can I use online resources to learn new things and stay entertained during social distancing?\",\n",
+      "\"What are some ways I can maintain healthier habits and avoid unhelpful ones when my schedule is disrupted?\",\n",
+      "\"How can I get involved in helping my community while practicing social distancing and good hygiene?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3597720: {\n",
+      "\"questions\": [\n",
+      "\"How does the way I think about a situation impact my feelings and behaviors?\",\n",
+      "\"What can challenging negative thoughts do to improve my mood and help me see new options?\",\n",
+      "\"Can you give me an example of how to reframe a negative thought to a more positive or productive one?\",\n",
+      "\"How can I better understand the situation more realistically and see the parts that I can control right now?\",\n",
+      "\"What can I do to focus on the things I can do, and make everything feel more manageable?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7535002: {\n",
+      "\"questions\": [\n",
+      "\"What strategies can I use to challenge common thinking traps?\",\n",
+      "\"How can I identify if I'm falling into distorted thinking patterns?\",\n",
+      "\"What is a helpful way to break a thinking trap by looking at the thoughts?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4134858: {\n",
+      "\"questions\": [\n",
+      "\"What are some common causes of grief outside of losing a person or pet?\",\n",
+      "\"How can I healthily express my feelings of grief?\",\n",
+      "\"Where can I find help from a mental health professional if I'm struggling with grief?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9676742: {\n",
+      "\"questions\": [\n",
+      "\"How can I find a family doctor using the College of Physicians and Surgeons of BC's Find a Physician tool?\",\n",
+      "\"Are walk-in clinics a good option for finding a regular doctor if I have ongoing care needs?\",\n",
+      "\"How can I obtain a referral to see a psychiatrist?\",\n",
+      "\"Where can I find a registered psychologist in BC?\",\n",
+      "\"How can I visit the BC Association of Clinical Counsellors to find a clinical counsellor?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2612846: {\n",
+      "\"questions\": [\n",
+      "\"Can I make an appointment with a psychiatrist directly or do I need a referral from a doctor?\",\n",
+      "\"What is the process for self-referring to a mental health program in BC?\",\n",
+      "\"How can I find my local mental health center in BC?\",\n",
+      "\"What services are available at a mental health center besides access to a psychiatrist?\",\n",
+      "\"How can I get help from HealthlinkBC if I need assistance finding a mental health service?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3166337: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find a Registered Psychologist?\",\n",
+      "\"Do I need a doctor's referral to see a psychologist?\",\n",
+      "\"How can I find out about lower-cost options for counselling in my community?\",\n",
+      "\"Are the costs of seeing a Registered Psychologist covered by MSP?\",\n",
+      "\"Can I ask for a sliding scale based on my income when making an appointment with a psychologist?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7009409: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find directories of counsellors?\",\n",
+      "\"Do I need a doctor's referral to see a counsellor?\",\n",
+      "\"How can I find out about lower-cost options to access counselling in my community?\",\n",
+      "\"Are counselling services at public mental health centers covered by MSP?\",\n",
+      "\"How can Indigenous community members access counselling services locally or through the First Nations Health Authority?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9630578: {\n",
+      "\"questions\": [\n",
+      "\"Where can I look for support groups specific to mental illnesses like mood disorders or post-traumatic stress disorder?\",\n",
+      "\"How can I find out about support groups offered by the Canadian Mental Health Association or the BC Schizophrenia Society?\",\n",
+      "\"Where can I find online support if I have trouble getting to an in-person support group?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1585622: {\n",
+      "\"questions\": [\n",
+      "\"Where can I encourage my child to talk about their mental health concerns?\",\n",
+      "\"Who can I talk to if I'm concerned about my child's mental health?\",\n",
+      "\"Where can I find specific child and youth mental health services and providers?\",\n",
+      "\"Where can I get guidance, information, and support for youth and parent mental health?\",\n",
+      "\"Where can young people find easy-to-access mental health care and support?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1833460: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find information about local mental health services for older adults?\",\n",
+      "\"Can I get information about senior's mental health resources from the Canadian Coalition for Seniors' Mental Health?\",\n",
+      "\"Who can I contact if I'm concerned about a loved one's mental health?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6981545: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find self-help resources for depression that are appropriate for teens?\",\n",
+      "\"Can you recommend some self-help materials for managing depression while pregnant or after delivery?\",\n",
+      "\"Are there any online self-help programs for depression that are based outside of BC but offer good information?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2115228: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find self-help resources for anxiety if I'm not in British Columbia?\",\n",
+      "\"Can I use Anxiety Canada's resources if I'm an adult but not a teen or young adult?\",\n",
+      "\"What resources does the BC Reproductive Mental Health Program offer for anxiety during and after birth?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2009922: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find free programs to help manage ongoing physical or mental health issues?\",\n",
+      "\"Are there any self-guided workbooks that discuss the links between physical health and mental health?\",\n",
+      "\"What resources are available to help me cope with depression, low mood, or anxiety caused by a chronic health condition?\",\n",
+      "\"Can I purchase any booklets that offer practical tips to cope with a chronic health problem?\",\n",
+      "\"Where can I find one-on-one support from a therapist or counselor if I'm having a hard time coping?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6167248: {\n",
+      "\"questions\": [\n",
+      "\"How can I determine if my substance use is causing problems?\",\n",
+      "\"Where can I find resources for helping someone else with their substance use?\",\n",
+      "\"What are some ways I can think about my relationship with substances?\",\n",
+      "\"What tips and resources are available for cutting back or quitting substance use?\",\n",
+      "\"How can I contact the Alcohol and Drug Information Referral Service for help in my area?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6779222: {\n",
+      "\"questions\": [\n",
+      "\"How can I find out if my BC Medical Services Plan covers mental health services like counselling or therapy?\",\n",
+      "\"Where can I find mental health clinics or hospitals that offer free or low-cost counselling services?\",\n",
+      "\"How can I find mental health services in my area using HealthLink BC’s Find Services tool and Locator app?\",\n",
+      "\"Where can I look for private mental health services that offer a sliding scale based on income?\",\n",
+      "\"How can mental health organizations like Anxiety Canada or the Canadian Mental Health Association help me find local mental health services or connections?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1546812: {\n",
+      "\"questions\": [\n",
+      "\"Is there a program to help me pay for my psychiatric medications?\",\n",
+      "\"How can I apply for the Psychiatric Medications Plan through BC PharmaCare?\",\n",
+      "\"How long does the coverage for Plan G typically last?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2176317: {\n",
+      "\"questions\": [\n",
+      "\"Are there any mental health services covered by MSP that I could access for free?\",\n",
+      "\"What are some ways I can offset the costs of mental health services not covered by MSP?\",\n",
+      "\"How can I find mental health providers that offer lower-cost services or a sliding scale based on income?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3284724: {\n",
+      "\"questions\": [\n",
+      "\"What are the different income levels for provincial income assistance?\",\n",
+      "\"How can mental illness be considered in PWD or PPMB income assistance applications?\",\n",
+      "\"Where can I find resources to help me navigate the application process for PWD and PPMB benefits?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 5981663: {\n",
+      "\"questions\": [\n",
+      "\"What is the purpose of MSP in British Columbia?\",\n",
+      "\"How are MSP fees determined and paid?\",\n",
+      "\"Where can I get help with applying for or managing my MSP coverage?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6623577: {\n",
+      "\"questions\": [\n",
+      "\"What does it mean when someone recommends me to another service in healthcare?\",\n",
+      "\"How do I access a specialist like a psychiatrist for mental health in BC?\",\n",
+      "\"What is the process for getting an appointment with a specialist after a referral from my doctor?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6851366: {\n",
+      "\"questions\": [\n",
+      "\"Are psychiatrists able to prescribe medications, while psychologists cannot?\",\n",
+      "\"How are the education and training different between psychiatrists and registered psychologists?\",\n",
+      "\"How does payment for services differ between accessing a psychologist in the public system versus the private system?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1898078: {\n",
+      "\"questions\": [\n",
+      "\"What are the commonalities between psychotherapy and counseling?\",\n",
+      "\"Can you explain the role of a professional therapist or counselor in the healing process?\",\n",
+      "\"What are some examples of different psychotherapies that have been found effective for various problems or illnesses?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1706961: {\n",
+      "\"questions\": [\n",
+      "\"What is the main focus of CBT in contrast to DBT?\",\n",
+      "\"How does DBT deals with emotions and relationships differently than CBT?\",\n",
+      "\"In what kind of mental health issues have DBT been proven to be effective?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4925221: {\n",
+      "\"questions\": [\n",
+      "\"What are the different classes of antidepressant medications and how do they work?\",\n",
+      "\"What are some common medications for each class of antidepressants?\",\n",
+      "\"What factors might affect how a medication works for different people?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1043721: {\n",
+      "\"questions\": [\n",
+      "\"How can I bring up my concerns about my diagnosis with the professional who provided it?\",\n",
+      "\"What is the process for getting a second opinion from a different professional?\",\n",
+      "\"How can I discuss my options and get a referral to a different program or health professional for a second opinion?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 8471978: {\n",
+      "\"questions\": [\n",
+      "\"What should I do if I don't think my prescribed antidepressant is working?\",\n",
+      "\"Who can I contact if I want to discuss the common side effects of my medication?\",\n",
+      "\"Is it safe to stop taking my psychiatric medication or change the dose without consulting a health care provider?\",\n",
+      "\"What should I do if I experience a serious allergic reaction to my medication?\",\n",
+      "\"How can I effectively communicate with my doctor about symptoms or side effects that I find embarrassing?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2746837: {\n",
+      "\"questions\": [\n",
+      "\"How can I help an adult in my life who is struggling with potential mental health issues?\",\n",
+      "\"What are the risks of diagnosing a health condition, especially online, without professional input?\",\n",
+      "\"What is the role of the person being treated in their own recovery and how can I support them in this process?\",\n",
+      "\"How can I effectively express my concerns and work together with the person to find a solution?\",\n",
+      "\"What should I know about the BC Mental Health Act in relation to getting help for someone who is at risk of harming themselves or others?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 5640861: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find a family doctor to talk about my concerns for my child or teenager?\",\n",
+      "\"What kind of services and support can I expect from a school counselor regarding my child's mental health?\",\n",
+      "\"What is the role of Kelty Mental Health Resource Centre and how can their Help Finder tool assist me in navigating the mental health system for my child?\",\n",
+      "\"Does the Canadian Mental Health Association’s BC Division offer any resources or programs for managing behavior problems in children ages 3 to 12?\",\n",
+      "\"Where can I find HeretoHelp resources on child and youth mental health and substance use?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3393534: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find more information about mental health problems that may arise from a brain injury?\",\n",
+      "\"Are there service providers who are knowledgeable about both brain injuries and mental health?\",\n",
+      "\"How can I find local and regional groups in BC that support those with brain injuries?\",\n",
+      "\"What organizations provide information and support for those affected by brain injuries in BC?\",\n",
+      "\"Are mental health challenges related to brain injuries always treatable?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3839472: {\n",
+      "\"questions\": [\n",
+      "\"How can I recognize and empathize with my loved one's experience with depression?\",\n",
+      "\"Where can I learn more about depression to better understand it?\",\n",
+      "\"How can I create a safe space for my loved one to talk about their depression?\",\n",
+      "\"What are some things I should avoid, like giving unsolicited advice?\",\n",
+      "\"How can I support my loved one in their treatment and recovery?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3055896: {\n",
+      "\"questions\": [\n",
+      "\"What are some trustworthy resources where I can learn more about schizophrenia?\",\n",
+      "\"How can I support my loved one during episodes of psychosis, especially when their hallucinations or delusions are distressing?\",\n",
+      "\"What is the role of a family member in helping a loved one manage their treatment for schizophrenia?\",\n",
+      "\"What can I do if my loved one refuses treatment for schizophrenia and I believe they are at risk of harming themselves or others?\",\n",
+      "\"What should I do if my loved one expresses thoughts of ending their life?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4200873: {\n",
+      "\"questions\": [\n",
+      "\"How can learning about my parent's mental illness make it easier for me to understand their experiences?\",\n",
+      "\"What self-care activities can I do to take care of myself while my parent is unwell?\",\n",
+      "\"Where can I find programs specifically designed for young people in my situation?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1337085: {\n",
+      "\"questions\": [\n",
+      "\"What resources are available for someone experiencing thoughts of suicide?\",\n",
+      "\"How can I help someone I care about who is talking about suicide?\",\n",
+      "\"What should I do in a situation where there is immediate danger or risk of harm due to suicide thoughts?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2447683: {\n",
+      "\"questions\": [\n",
+      "\"What are the potential benefits and harms of using cannabis?\",\n",
+      "\"How can the timing of when a person starts using cannabis impact potential harms?\",\n",
+      "\"Are there certain groups of people who should avoid using cannabis due to increased risk of psychosis or schizophrenia?\",\n",
+      "\"How can mixing drugs like cannabis and alcohol increase the possibility of experiencing harms?\",\n",
+      "\"What is a simple way to gauge the use of psychoactive substances and prevent potential harms?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7995219: {\n",
+      "\"questions\": [\n",
+      "\"Can you explain more about what mental well-being entails?\",\n",
+      "\"How would you define a mental illness and how does it affect a person's life?\",\n",
+      "\"Can a person have good mental health even if they have been diagnosed with a mental illness?\",\n",
+      "\"How does mental health exist on a continuum, similar to physical health?\",\n",
+      "\"What supports and tools can help someone live well and find meaning despite mental health challenges?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 2903196: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find info sheets on mood disorders and bipolar disorder?\",\n",
+      "\"Do you have a self-test for depression that also screens for signs of bipolar disorder?\",\n",
+      "\"Can I find a series to help me deal with a diagnosis of depression and work with my doctor?\",\n",
+      "\"Are there any books or resources with personal stories about depression?\",\n",
+      "\"How can I access the What is Depression? booklet with audio in plain language for lower literacy readers?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4893447: {\n",
+      "\"questions\": [\n",
+      "\"How can I distinguish between normal anxiety and an anxiety disorder?\",\n",
+      "\"What are some examples of how anxiety disorders can impact someone's daily life?\",\n",
+      "\"What are the differences between the former and current classifications of some anxiety disorders?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 8612349: {\n",
+      "\"questions\": [\n",
+      "\"What are the usual causes of stress?\",\n",
+      "\"How is anxiety different from stress in terms of what it focuses on?\",\n",
+      "\"Are the physical sensations of stress and anxiety always similar?\",\n",
+      "\"Can anxiety be triggered by a specific threat, or is it always about vague worries?\",\n",
+      "\"How can stress and anxiety impact well-being and daily life if they last for a long time?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6062232: {\n",
+      "\"questions\": [\n",
+      "\"How is sadness different from the illness of depression?\",\n",
+      "\"Can feelings of sadness impact daily life in a significant way like depression does?\",\n",
+      "\"Are there any physical symptoms associated with depression that don't occur with normal sadness?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9045344: {\n",
+      "\"questions\": [\n",
+      "\"How does persistent depressive disorder (PDD) differ from major depressive disorder in terms of symptoms and duration?\",\n",
+      "\"What are the symptoms of PDD?\",\n",
+      "\"Why is it a misconception that PDD is less severe than major depressive disorder?\",\n",
+      "\"How can PDD affect a person's ability to work and enjoy hobbies?\",\n",
+      "\"Why might someone with PDD think it is just part of their personality?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9020440: {\n",
+      "\"questions\": [\n",
+      "\"What is the relationship between cyclothymic disorder and bipolar disorder?\",\n",
+      "\"What are the symptoms of hypomania and depression associated with cyclothymic disorder?\",\n",
+      "\"How does the frequency of symptoms in cyclothymic disorder compare to bipolar disorder?\",\n",
+      "\"What wellness period are people with cyclothymic disorder likely to experience?\",\n",
+      "\"Who should I consult if I suspect I have cyclothymic disorder?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 5819325: {\n",
+      "\"questions\": [\n",
+      "\"What does it mean when someone experiences four or more episodes of depression and/or mania in one year?\",\n",
+      "\"Who can experience rapid cycling and when does it typically occur?\",\n",
+      "\"Is rapid cycling a permanent condition for those diagnosed with bipolar disorder?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4816493: {\n",
+      "\"questions\": [\n",
+      "\"Where can I find more information about the five groups of symptoms for borderline personality disorder?\",\n",
+      "\"Do you know of any support groups or resources for borderline personality disorder in cities other than Vancouver and Victoria?\",\n",
+      "\"Can you recommend any helpful books for managing borderline personality disorder?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9178453: {\n",
+      "\"questions\": [\n",
+      "\"How does schizoid personality disorder affect social interactions and relationships?\",\n",
+      "\"What is the difference between schizoid personality disorder and schizophrenia?\",\n",
+      "\"How is schizoid personality disorder typically treated and managed?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9679704: {\n",
+      "\"questions\": [\n",
+      "\"How does antisocial personality disorder affect a person's thoughts and behaviors?\",\n",
+      "\"What is the relationship between conduct disorder and antisocial personality disorder?\",\n",
+      "\"How can treatment help someone with antisocial personality disorder?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9167296: {\n",
+      "\"questions\": [\n",
+      "\"How does a personality disorder, such as obsessive-compulsive personality disorder, affect a person's daily life and thoughts?\",\n",
+      "\"What are the key differences between obsessive-compulsive personality disorder and obsessive-compulsive disorder?\",\n",
+      "\"How is treatment typically approached for individuals with obsessive-compulsive personality disorder?\",\n",
+      "\"What kind of distress might a person with obsessive-compulsive personality disorder experience when they can't achieve control or perfectionism?\",\n",
+      "\"Where can someone seek help for obsessive-compulsive personality disorder in British Columbia?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 9049618: {\n",
+      "\"questions\": [\n",
+      "\"What are the main symptoms of binge-eating disorder?\",\n",
+      "\"How can I recognize if someone has binge-eating disorder?\",\n",
+      "\"What makes binge-eating disorder different from just overeating?\",\n",
+      "\"How is binge-eating disorder treated?\",\n",
+      "\"Who should I consult if I think I have binge-eating disorder?\"]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7984793: {\n",
+      "\"questions\": [\n",
+      "\"How does the 'split mind' in schizophrenia differ from the fragmented identity in dissociative identity disorder?\",\n",
+      "\"What is the significance of dissociation in dissociative identity disorder?\",\n",
+      "\"How does schizophrenia affect a person's perception of reality differently than dissociative identity disorder?\",\n",
+      "\"What are some common misconceptions about dissociative identity disorder, such as its portrayal in media, and how does it contrast with the actual experience of those diagnosed with it?\",\n",
+      "\"How does schizophrenia impact an individual's thoughts, speech, and emotional expression as opposed to dissociative identity disorder?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6085633: {\n",
+      "\"questions\": [\n",
+      "\"What is the difference between psychosis and schizophrenia in terms of their definitions?\",\n",
+      "\"What are some common symptoms of psychosis, such as hallucinations and delusions?\",\n",
+      "\"What mental illnesses or factors, aside from schizophrenia, can cause psychosis?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 1896541: {\n",
+      "\"questions\": [\n",
+      "\"What do positive symptoms of schizophrenia involve?\",\n",
+      "\"Can you explain what negative symptoms of schizophrenia are?\",\n",
+      "\"What are cognitive symptoms and how do they relate to schizophrenia?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4962901: {\n",
+      "\"questions\": [\n",
+      "\"Can you explain what a prodrome is in simpler terms?\",\n",
+      "\"What are some early signs or symptoms of psychosis?\",\n",
+      "\"Why is it important to seek help early if you or a loved one is experiencing symptoms of a mental health problem?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6869967: {\n",
+      "\"questions\": [\n",
+      "\"How can I find a healthcare professional who specializes in assessing and treating ADHD in adults?\",\n",
+      "\"What resources and support are available for adults with ADHD in Canada?\",\n",
+      "\"Is there a support group for adults with ADHD in the Vancouver area?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4824231: {\n",
+      "\"questions\": [\n",
+      "\"What does it mean when someone says they are addicted to something like shopping?\",\n",
+      "\"How does the evidence suggest substance use and addiction differ?\",\n",
+      "\"Can you explain how substance dependence develops and what it involves?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4294616: {\n",
+      "\"questions\": [\n",
+      "\"What are some ways I can get help for an alcohol or drug use problem?\",\n",
+      "\"How does my health and social situation factor into finding the right help for me?\",\n",
+      "\"Can I talk to a nurse or doctor confidentially about drug use and what resources are available in my area?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4373204: {\n",
+      "\"questions\": [\n",
+      "\"How can I determine if I'm drinking too much considering my unique relationship with alcohol?\",\n",
+      "\"What are the signs of alcohol use becoming a problem in my life or in the lives of others?\",\n",
+      "\"Where can I get personalized feedback about my drinking pattern as compared to Canada's Low-Risk Alcohol Drinking Guidelines?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 7807643: {\n",
+      "\"questions\": [\n",
+      "\"If cannabis smoke contains harmful toxins, why is the risk of cancer less for cannabis smokers than tobacco smokers?\",\n",
+      "\"How do the negative effects of cannabis on the brain compare to those of alcohol?\",\n",
+      "\"What are the benefits of legalizing cannabis in terms of minimizing potential harms?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 4352464: {\n",
+      "\"questions\": [\n",
+      "\"How can I help my child develop skills to make good choices regarding drugs?\",\n",
+      "\"What are some ways to strengthen my relationship with my child through discussions about drugs?\",\n",
+      "\"How can I support my child in developing critical thinking skills about drug use?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 6521784: {\n",
+      "\"questions\": [\n",
+      "\"What are the medical conditions that CBD has been used in the treatment of?\",\n",
+      "\"How does CBD produce therapeutic effects in the body?\",\n",
+      "\"Are there any recorded negative drug interactions with CBD?\",\n",
+      "\"What are some potential side effects of using CBD?\",\n",
+      "\"Is there any research on the use of CBD for Alzheimer's disease?\"\n",
+      "]\n",
+      "}\n",
+      "Raw JSON data for doc_id 3221856: {\n",
+      "\"questions\": [\n",
+      "\"What are the potential harms of vaping?\",\n",
+      "\"How is nicotine delivered through vaping?\",\n",
+      "\"What is the impact of vaping marketing strategies on young people?\"\n",
+      "]\n",
+      "}\n"
+     ]
+    }
+   ],
+   "source": [
+    "results={}\n",
+    "for doc in tqdm(documents):\n",
+    "    doc_id=doc['Question_ID']\n",
+    "    if doc_id in results:\n",
+    "        continue\n",
+    "    questions_raw = generate_data(doc)\n",
+    "    print(f\"Raw JSON data for doc_id {doc_id}: {questions_raw}\")\n",
+    "    \n",
+    "    # Check if it's empty\n",
+    "    if not questions_raw:\n",
+    "        print(f\"Error: Empty response for doc_id {doc_id}\")\n",
+    "    else:\n",
+    "        # Try parsing it\n",
+    "        try:\n",
+    "            questions = json.loads(questions_raw)\n",
+    "            results[doc_id] = questions['questions']\n",
+    "        except json.JSONDecodeError as e:\n",
+    "            print(f\"JSON decoding failed for doc_id {doc_id}: {e}\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 123,
+   "id": "4bbea76c-e98f-42a9-b2e0-8296828a5e24",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "final_results = []\n",
+    "\n",
+    "for doc_id, questions in results.items():\n",
+    "    for q in questions:\n",
+    "        final_results.append((doc_id, q))\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 128,
+   "id": "2b72649f-4b4b-402a-8d83-310009aaa294",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(6361820,\n",
+       " 'What signs should parents and caretakers look out for in children under 18 that may indicate a mental health condition?')"
+      ]
+     },
+     "execution_count": 128,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "final_results[10]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 129,
+   "id": "20b5f107-951d-4842-895e-f1100e16e2a2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_results = pd.DataFrame(final_results, columns=['id', 'question'])\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 131,
+   "id": "c72c8a72-db39-4019-91ea-9444d7f20523",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>id</th>\n",
+       "      <th>question</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>1590140</td>\n",
+       "      <td>How do mental illnesses affect a person's dail...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>1590140</td>\n",
+       "      <td>What are some examples of serious mental illne...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>1590140</td>\n",
+       "      <td>Why is it a misconception to associate mental ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>1590140</td>\n",
+       "      <td>How are mental illnesses treated, and what is ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>1590140</td>\n",
+       "      <td>Can a person with mental illness become indepe...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>2110618</td>\n",
+       "      <td>Who are the groups most commonly affected by d...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>2110618</td>\n",
+       "      <td>How do mental illnesses affect children and th...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>2110618</td>\n",
+       "      <td>What behaviors should parents and caretakers w...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>6361820</td>\n",
+       "      <td>How does mental illness affect different popul...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>6361820</td>\n",
+       "      <td>At what age are individuals most susceptible t...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "        id                                           question\n",
+       "0  1590140  How do mental illnesses affect a person's dail...\n",
+       "1  1590140  What are some examples of serious mental illne...\n",
+       "2  1590140  Why is it a misconception to associate mental ...\n",
+       "3  1590140  How are mental illnesses treated, and what is ...\n",
+       "4  1590140  Can a person with mental illness become indepe...\n",
+       "5  2110618  Who are the groups most commonly affected by d...\n",
+       "6  2110618  How do mental illnesses affect children and th...\n",
+       "7  2110618  What behaviors should parents and caretakers w...\n",
+       "8  6361820  How does mental illness affect different popul...\n",
+       "9  6361820  At what age are individuals most susceptible t..."
+      ]
+     },
+     "execution_count": 131,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df_results.head(10)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 133,
+   "id": "e2cb78c2-6186-4aa9-93f3-cd501b9dcd24",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df_results.to_csv('../dataset/ground_truth_data.csv', index=False)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/minsearch.py

@@ -0,0 +1,96 @@
+import pandas as pd
+
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.metrics.pairwise import cosine_similarity
+
+import numpy as np
+
+
+class Index:
+    """
+    A simple search index using TF-IDF and cosine similarity for text fields and exact matching for keyword fields.
+
+    Attributes:
+        text_fields (list): List of text field names to index.
+        keyword_fields (list): List of keyword field names to index.
+        vectorizers (dict): Dictionary of TfidfVectorizer instances for each text field.
+        keyword_df (pd.DataFrame): DataFrame containing keyword field data.
+        text_matrices (dict): Dictionary of TF-IDF matrices for each text field.
+        docs (list): List of documents indexed.
+    """
+
+    def __init__(self, text_fields, keyword_fields, vectorizer_params={}):
+        """
+        Initializes the Index with specified text and keyword fields.
+
+        Args:
+            text_fields (list): List of text field names to index.
+            keyword_fields (list): List of keyword field names to index.
+            vectorizer_params (dict): Optional parameters to pass to TfidfVectorizer.
+        """
+        self.text_fields = text_fields
+        self.keyword_fields = keyword_fields
+
+        self.vectorizers = {field: TfidfVectorizer(**vectorizer_params) for field in text_fields}
+        self.keyword_df = None
+        self.text_matrices = {}
+        self.docs = []
+
+    def fit(self, docs):
+        """
+        Fits the index with the provided documents.
+
+        Args:
+            docs (list of dict): List of documents to index. Each document is a dictionary.
+        """
+        self.docs = docs
+        keyword_data = {field: [] for field in self.keyword_fields}
+
+        for field in self.text_fields:
+            texts = [doc.get(field, '') for doc in docs]
+            self.text_matrices[field] = self.vectorizers[field].fit_transform(texts)
+
+        for doc in docs:
+            for field in self.keyword_fields:
+                keyword_data[field].append(doc.get(field, ''))
+
+        self.keyword_df = pd.DataFrame(keyword_data)
+
+        return self
+
+    def search(self, query, filter_dict={}, boost_dict={}, num_results=10):
+        """
+        Searches the index with the given query, filters, and boost parameters.
+
+        Args:
+            query (str): The search query string.
+            filter_dict (dict): Dictionary of keyword fields to filter by. Keys are field names and values are the values to filter by.
+            boost_dict (dict): Dictionary of boost scores for text fields. Keys are field names and values are the boost scores.
+            num_results (int): The number of top results to return. Defaults to 10.
+
+        Returns:
+            list of dict: List of documents matching the search criteria, ranked by relevance.
+        """
+        query_vecs = {field: self.vectorizers[field].transform([query]) for field in self.text_fields}
+        scores = np.zeros(len(self.docs))
+
+        # Compute cosine similarity for each text field and apply boost
+        for field, query_vec in query_vecs.items():
+            sim = cosine_similarity(query_vec, self.text_matrices[field]).flatten()
+            boost = boost_dict.get(field, 1)
+            scores += sim * boost
+
+        # Apply keyword filters
+        for field, value in filter_dict.items():
+            if field in self.keyword_fields:
+                mask = self.keyword_df[field] == value
+                scores = scores * mask.to_numpy()
+
+        # Use argpartition to get top num_results indices
+        top_indices = np.argpartition(scores, -num_results)[-num_results:]
+        top_indices = top_indices[np.argsort(-scores[top_indices])]
+
+        # Filter out zero-score results
+        top_docs = [self.docs[i] for i in top_indices if scores[i] > 0]
+
+        return top_docs

src/app.py

@@ -0,0 +1,230 @@
+import streamlit as st
+import time
+import uuid
+from datetime import datetime
+from zoneinfo import ZoneInfo
+import pandas as pd
+
+import rag
+from db import (
+    save_conversation,
+    save_feedback,
+    get_recent_conversations,
+    get_feedback_stats,
+    init_db,
+    verify_conversation_saved
+)
+import logging
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+tz = ZoneInfo("Europe/Berlin")
+
+def print_log(message):
+    print(message, flush=True)
+
+def main():
+    print_log("Starting the Mental Health Assistant application")
+    st.title("Mental Health Assistant")
+
+    # Initialize the database
+    init_db()
+
+    # Session state initialization
+    if "conversation_id" not in st.session_state:
+        st.session_state.conversation_id = str(uuid.uuid4())
+        print_log(f"New conversation started with ID: {st.session_state.conversation_id}")
+    if "count" not in st.session_state:
+        st.session_state.count = 0
+        print_log("Feedback count initialized to 0")
+    if "feedback_given" not in st.session_state:
+        st.session_state.feedback_given = False
+    if "past_questions" not in st.session_state:
+        st.session_state.past_questions = []
+    if "chat_history" not in st.session_state:
+        st.session_state.chat_history = []  # Stores the questions and answers for display
+    if "clear_chat" not in st.session_state:
+        st.session_state.clear_chat = False
+
+    # Session state for user input
+    if "user_input" not in st.session_state:
+        st.session_state.user_input = ""  # This will control the text input field value
+
+    # Check if we need to clear the chat
+    if st.session_state.clear_chat:
+        st.session_state.chat_history = []
+        st.session_state.clear_chat = False
+
+    # Model selection
+    model_choice = st.selectbox(
+        "Select a model:",
+        ["gemma2-9b-it", "llama-3.1-70b-versatile", "llama3-70b-8192", "mixtral-8x7b-32768"],
+    )
+    print_log(f"User selected model: {model_choice}")
+
+    # User input text box
+    user_input = st.text_input("Ask a question about mental health:", value=st.session_state.user_input)
+
+    if "last_conversation_id" not in st.session_state:
+        st.session_state.last_conversation_id = None
+
+    if st.button("Ask"):
+        logger.debug(f"Ask button pressed. User input: {user_input}")
+
+        # Check if input is valid
+        if user_input.strip() == "":
+            st.warning("Please enter a question before asking.")
+        elif user_input in st.session_state.past_questions:
+            st.warning("You've already asked this question.")
+        else:
+            # Proceed with getting an answer from the assistant
+            print_log(f"User asked: '{user_input}'")
+            with st.spinner("Processing..."):
+                print_log(f"Getting answer from assistant using {model_choice} model")
+                start_time = time.time()
+                answer_data = rag.rag(user_input, model=model_choice)
+                end_time = time.time()
+                print_log(f"Answer received in {end_time - start_time:.2f} seconds")
+                st.success("Completed!")
+                st.write(answer_data["answer"])
+
+                # Store the conversation in chat history
+                st.session_state.chat_history.append({
+                "question": user_input,
+                "answer": answer_data["answer"],
+                "relevance": answer_data["relevance"],
+                "model": answer_data["model_used"]})
+
+                # Display monitoring information
+                st.write(f"Response time: {answer_data['response_time']:.2f} seconds")
+                st.write(f"Relevance: {answer_data['relevance']}")
+                st.write(f"Model used: {answer_data['model_used']}")
+                st.write(f"Total tokens: {answer_data['total_tokens']}")
+
+                # Save conversation to database
+                logger.debug(f"Attempting to save conversation: {st.session_state.conversation_id}")
+                save_conversation(st.session_state.conversation_id, user_input, answer_data)
+                logger.debug(f"Conversation saved. Verifying...")
+                verify_conversation_saved(st.session_state.conversation_id)
+
+
+
+                # Update the last_conversation_id and reset feedback
+                st.session_state.last_conversation_id = st.session_state.conversation_id
+                st.session_state.conversation_id = str(uuid.uuid4())  # New conversation ID for the next question
+                st.session_state.feedback_given = False  # Reset feedback state
+                st.session_state.past_questions.append(user_input)  # Add the question to past questions
+
+                # Clear the input field by resetting session state variable
+                st.session_state.user_input = ""  # Reset input for next question
+
+    # Feedback buttons
+    col1, col2 = st.columns(2)
+    with col1:
+        if st.button("+1"):
+            if st.session_state.last_conversation_id and not st.session_state.feedback_given:
+                save_feedback(st.session_state.last_conversation_id, 1)
+                st.success("Positive feedback saved!")
+                st.session_state.feedback_given = True  # Mark feedback as given
+                st.session_state.last_conversation_id = None  # Clear last conversation
+                st.session_state.clear_chat = True  # Set flag to clear chat on next rerun
+                st.rerun()  # Rerun the app to refresh the UI
+            elif st.session_state.feedback_given:
+                st.warning("Feedback has already been provided for this conversation.")
+            else:
+                st.warning("No conversation to provide feedback for.")
+    with col2:
+        if st.button("-1"):
+            if st.session_state.last_conversation_id and not st.session_state.feedback_given:
+                save_feedback(st.session_state.last_conversation_id, -1)
+                st.success("Negative feedback saved!")
+                st.session_state.feedback_given = True  # Mark feedback as given
+                st.session_state.last_conversation_id = None  # Clear last conversation
+                st.session_state.clear_chat = True  # Set flag to clear chat on next rerun
+                st.rerun()  # Rerun the app to refresh the UI
+            elif st.session_state.feedback_given:
+                st.warning("Feedback has already been provided for this conversation.")
+            else:
+                st.warning("No conversation to provide feedback for.")
+
+    # Display feedback status
+    if st.session_state.feedback_given:
+        st.info("Feedback has already been provided for this conversation.")
+    else:
+        st.info("You can provide feedback for the current conversation.")
+
+    # Display chat history
+    if st.session_state.chat_history:
+        st.subheader("Chat History")
+        for chat in st.session_state.chat_history:
+            st.write(f"**Q:** {chat['question']}")
+            st.write(f"**A:** {chat['answer']}")
+            st.write(f"*Relevance: {chat['relevance']}, Model: {chat['model']}*")
+            st.write("---")
+
+    # Display statistics dashboard
+    st.subheader("Statistics Dashboard")
+
+    # Calculate relevance stats
+    relevance_counts = {
+        "RELEVANT": 0,
+        "PARTLY_RELEVANT": 0,
+        "NON_RELEVANT": 0
+    }
+    recent_conversations = get_recent_conversations(limit=100)  # Get more conversations for better stats
+    for conv in recent_conversations:
+        if conv['relevance'] in relevance_counts:
+            relevance_counts[conv['relevance']] += 1
+
+    # Display relevance stats
+    st.write("Response Relevance Distribution")
+    relevance_df = pd.DataFrame({
+        'Relevance': list(relevance_counts.keys()),
+        'Count': list(relevance_counts.values())
+    })
+    st.bar_chart(relevance_df.set_index('Relevance'))
+
+    # Display feedback stats
+    feedback_stats = get_feedback_stats()
+    st.write("User Feedback Distribution")
+    feedback_df = pd.DataFrame({
+        'Feedback': ['Positive', 'Negative'],
+        'Count': [feedback_stats['thumbs_up'], feedback_stats['thumbs_down']]
+    })
+    st.bar_chart(feedback_df.set_index('Feedback'))
+
+
+
+    # Display recent conversations
+    st.subheader("Recent Conversations")
+    relevance_filter = st.selectbox(
+        "Filter by relevance:",
+        ["All", "RELEVANT", "PARTLY_RELEVANT", "NON_RELEVANT"]
+    )
+    recent_conversations = get_recent_conversations(
+        limit=3,
+        relevance=relevance_filter if relevance_filter != "All" else None
+    )
+    for conv in recent_conversations:
+        st.write(f"Q: {conv['question']}")
+        st.write(f"A: {conv['answer']}")
+        st.write(f"Relevance: {conv['relevance']}")
+        st.write(f"Model: {conv['model_used']}")
+        st.write("---")
+
+    # Display feedback stats
+    feedback_stats = get_feedback_stats()
+    st.subheader("Feedback Statistics")
+    st.write(f"Thumbs up: {feedback_stats['thumbs_up']}")
+    st.write(f"Thumbs down: {feedback_stats['thumbs_down']}")
+
+    # Generate a new conversation ID for the next question
+    st.session_state.conversation_id = str(uuid.uuid4())
+
+print_log("Streamlit app loop completed")
+
+if __name__ == "__main__":
+    print_log("Mental Health Assistant application started")
+    main()

src/db.py

@@ -0,0 +1,322 @@
+import os
+import psycopg2
+from psycopg2.extras import DictCursor
+from datetime import datetime, timezone
+from zoneinfo import ZoneInfo
+import logging
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+RUN_TIMEZONE_CHECK = os.getenv('RUN_TIMEZONE_CHECK', '1') == '1'
+
+TZ_INFO = os.getenv("TZ", "Europe/Berlin")
+tz = ZoneInfo(TZ_INFO)
+
+
+def get_db_connection():
+    try:
+        conn = psycopg2.connect(
+            host=os.getenv("POSTGRES_HOST", "postgres"),
+            database=os.getenv("POSTGRES_DB", "mental_health"),
+            user=os.getenv("POSTGRES_USER", "newton"),
+            password=os.getenv("POSTGRES_PASSWORD", "Admin"),
+        )
+        logger.info("Successfully connected to the database")
+        return conn
+    except Exception as e:
+        logger.error(f"Error connecting to the database: {e}")
+        raise
+
+
+def init_db():
+    conn = get_db_connection()
+    try:
+        with conn.cursor() as cur:
+            # Check if tables exist
+            cur.execute("SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = 'conversations')")
+            conversations_exists = cur.fetchone()[0]
+            cur.execute("SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = 'feedback')")
+            feedback_exists = cur.fetchone()[0]
+            
+            if not conversations_exists:
+                logger.info("Creating conversations table")
+                cur.execute("""
+                    CREATE TABLE conversations (
+                        id TEXT PRIMARY KEY,
+                        question TEXT NOT NULL,
+                        answer TEXT NOT NULL,
+                        model_used TEXT NOT NULL,
+                        response_time FLOAT NOT NULL,
+                        relevance TEXT NOT NULL,
+                        relevance_explanation TEXT NOT NULL,
+                        prompt_tokens INTEGER NOT NULL,
+                        completion_tokens INTEGER NOT NULL,
+                        total_tokens INTEGER NOT NULL,
+                        eval_prompt_tokens INTEGER NOT NULL,
+                        eval_completion_tokens INTEGER NOT NULL,
+                        eval_total_tokens INTEGER NOT NULL,
+                        timestamp TIMESTAMP WITH TIME ZONE NOT NULL
+                    )
+                """)
+            
+            if not feedback_exists:
+                logger.info("Creating feedback table")
+                cur.execute("""
+                    CREATE TABLE feedback (
+                        id SERIAL PRIMARY KEY,
+                        conversation_id TEXT REFERENCES conversations(id),
+                        feedback INTEGER NOT NULL,
+                        timestamp TIMESTAMP WITH TIME ZONE NOT NULL
+                    )
+                """)
+            # Add comments table
+            cur.execute("SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_name = 'comments')")
+            comments_exists = cur.fetchone()[0]
+            
+            if not comments_exists:
+                logger.info("Creating comments table")
+                cur.execute("""
+                    CREATE TABLE comments (
+                        id SERIAL PRIMARY KEY,
+                        conversation_id TEXT REFERENCES conversations(id),
+                        comment TEXT NOT NULL,
+                        timestamp TIMESTAMP WITH TIME ZONE NOT NULL
+                    )
+                """)
+            
+            conn.commit()
+            logger.info("Database tables initialized successfully")
+    except Exception as e:
+        logger.error(f"Error initializing database: {e}")
+        conn.rollback()
+    finally:
+        conn.close()
+
+
+def save_conversation(conversation_id, question, answer_data, timestamp=None):
+    if timestamp is None:
+        timestamp = datetime.now(tz)
+
+    conn = get_db_connection()
+    try:
+        with conn.cursor() as cur:
+            logger.debug(f"Saving conversation: {conversation_id}")
+            logger.debug(f"Question: {question}")
+            logger.debug(f"Answer data: {answer_data}")
+            cur.execute(
+                """
+                INSERT INTO conversations 
+                (id, question, answer, model_used, response_time, relevance, 
+                relevance_explanation, prompt_tokens, completion_tokens, total_tokens, 
+                eval_prompt_tokens, eval_completion_tokens, eval_total_tokens, timestamp)
+                VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                """,
+                (
+                    conversation_id,
+                    question,
+                    answer_data["answer"],
+                    answer_data["model_used"],
+                    answer_data["response_time"],
+                    answer_data["relevance"].upper(),  # Ensure it's uppercase
+                    answer_data["relevance_explanation"],
+                    answer_data["prompt_tokens"],
+                    answer_data["completion_tokens"],
+                    answer_data["total_tokens"],
+                    answer_data["eval_prompt_tokens"],
+                    answer_data["eval_completion_tokens"],
+                    answer_data["eval_total_tokens"],
+                    timestamp
+                ),
+            )
+        conn.commit()
+        logger.info(f"Successfully saved conversation: {conversation_id}")
+    except Exception as e:
+        logger.error(f"Error saving conversation: {e}")
+        conn.rollback()
+    finally:
+        conn.close()
+
+
+def save_feedback(conversation_id, feedback, timestamp=None):
+    if timestamp is None:
+        timestamp = datetime.now(tz)
+
+    conn = get_db_connection()
+    try:
+        with conn.cursor() as cur:
+            # Check if the conversation exists
+            cur.execute("SELECT id FROM conversations WHERE id = %s", (conversation_id,))
+            if cur.fetchone() is None:
+                logger.warning(f"Attempted to save feedback for non-existent conversation: {conversation_id}")
+                return  # Or handle this case as appropriate for your application
+
+            logger.info(f"Attempting to save feedback: conversation_id={conversation_id}, feedback={feedback}, timestamp={timestamp}")
+            cur.execute(
+                "INSERT INTO feedback (conversation_id, feedback, timestamp) VALUES (%s, %s, %s)",
+                (conversation_id, feedback, timestamp),
+            )
+        conn.commit()
+        logger.info(f"Feedback saved successfully for conversation {conversation_id}")
+    except Exception as e:
+        logger.error(f"Error saving feedback: {e}")
+        conn.rollback()
+        raise
+    finally:
+        conn.close()
+
+def save_comment(conversation_id, comment, timestamp=None):
+    if timestamp is None:
+        timestamp = datetime.now(tz)
+
+    conn = get_db_connection()
+    try:
+        with conn.cursor() as cur:
+            cur.execute(
+                "INSERT INTO comments (conversation_id, comment, timestamp) VALUES (%s, %s, %s)",
+                (conversation_id, comment, timestamp)
+            )
+        conn.commit()
+        logger.info(f"Comment saved successfully for conversation {conversation_id}")
+    except Exception as e:
+        logger.error(f"Error saving comment: {e}")
+        conn.rollback()
+    finally:
+        conn.close()
+
+def get_comments(conversation_id):
+    conn = get_db_connection()
+    try:
+        with conn.cursor(cursor_factory=DictCursor) as cur:
+            cur.execute(
+                "SELECT * FROM comments WHERE conversation_id = %s ORDER BY timestamp DESC",
+                (conversation_id,)
+            )
+            return cur.fetchall()
+    finally:
+        conn.close()
+def get_recent_conversations(limit=5, relevance=None):
+    conn = get_db_connection()
+    try:
+        with conn.cursor(cursor_factory=DictCursor) as cur:
+            query = """
+                SELECT c.*, f.feedback
+                FROM conversations c
+                LEFT JOIN feedback f ON c.id = f.conversation_id
+            """
+            params = []
+            if relevance:
+                if relevance in ["RELEVANT", "PARTLY_RELEVANT", "NON_RELEVANT"]:
+                    query += " WHERE c.relevance = %s"
+                    params.append(relevance)
+                else:
+                    logger.warning(f"Invalid relevance filter: {relevance}")
+            query += " ORDER BY c.timestamp DESC LIMIT %s"
+            params.append(limit)
+
+            logger.debug(f"Executing query: {query}")
+            logger.debug(f"Query parameters: {params}")
+            cur.execute(query, tuple(params))
+            return cur.fetchall()
+    except Exception as e:
+        logger.error(f"Error in get_recent_conversations: {e}")
+        return []
+    finally:
+        conn.close()
+
+
+def get_feedback_stats():
+    conn = get_db_connection()
+    try:
+        with conn.cursor(cursor_factory=DictCursor) as cur:
+            cur.execute("""
+                SELECT 
+                    SUM(CASE WHEN feedback > 0 THEN 1 ELSE 0 END) as thumbs_up,
+                    SUM(CASE WHEN feedback < 0 THEN 1 ELSE 0 END) as thumbs_down
+                FROM feedback
+            """)
+            return cur.fetchone()
+    finally:
+        conn.close()
+
+
+def check_timezone():
+    conn = get_db_connection()
+    try:
+        with conn.cursor() as cur:
+            cur.execute("SHOW timezone;")
+            db_timezone = cur.fetchone()[0]
+            print(f"Database timezone: {db_timezone}")
+
+            cur.execute("SELECT current_timestamp;")
+            db_time_utc = cur.fetchone()[0]
+            print(f"Database current time (UTC): {db_time_utc}")
+
+            db_time_local = db_time_utc.astimezone(tz)
+            print(f"Database current time ({TZ_INFO}): {db_time_local}")
+
+            py_time = datetime.now(tz)
+            print(f"Python current time: {py_time}")
+
+            # Use py_time instead of tz for insertion
+            cur.execute("""
+                INSERT INTO conversations 
+                (id, question, answer, model_used, response_time, relevance, 
+                relevance_explanation, prompt_tokens, completion_tokens, total_tokens, 
+                eval_prompt_tokens, eval_completion_tokens, eval_total_tokens, timestamp)
+                VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                RETURNING timestamp;
+            """, 
+            ('test', 'test question', 'test answer', 'test model', 0.0, 0.0, 
+             'test explanation', 0, 0, 0, 0, 0, 0, 0.0, py_time))
+
+            inserted_time = cur.fetchone()[0]
+            print(f"Inserted time (UTC): {inserted_time}")
+            print(f"Inserted time ({TZ_INFO}): {inserted_time.astimezone(tz)}")
+
+            cur.execute("SELECT timestamp FROM conversations WHERE id = 'test';")
+            selected_time = cur.fetchone()[0]
+            print(f"Selected time (UTC): {selected_time}")
+            print(f"Selected time ({TZ_INFO}): {selected_time.astimezone(tz)}")
+
+            # Clean up the test entry
+            cur.execute("DELETE FROM conversations WHERE id = 'test';")
+            conn.commit()
+    except Exception as e:
+        print(f"An error occurred: {e}")
+        conn.rollback()
+    finally:
+        conn.close()
+
+def verify_conversation_saved(conversation_id):
+    conn = get_db_connection()
+    try:
+        with conn.cursor() as cur:
+            cur.execute("SELECT * FROM conversations WHERE id = %s", (conversation_id,))
+            result = cur.fetchone()
+            if result:
+                logger.info(f"Verified conversation saved: {conversation_id}")
+            else:
+                logger.error(f"Failed to verify conversation saved: {conversation_id}")
+    except Exception as e:
+        logger.error(f"Error verifying conversation: {e}")
+    finally:
+        conn.close()
+def verify_feedback_saved(conversation_id):
+    conn = get_db_connection()
+    try:
+        with conn.cursor() as cur:
+            cur.execute("SELECT * FROM feedback WHERE conversation_id = %s", (conversation_id,))
+            result = cur.fetchone()
+            if result:
+                logger.info(f"Verified feedback saved for conversation: {conversation_id}")
+            else:
+                logger.error(f"Failed to verify feedback saved for conversation: {conversation_id}")
+    except Exception as e:
+        logger.error(f"Error verifying feedback: {e}")
+    finally:
+        conn.close()
+
+if RUN_TIMEZONE_CHECK:
+    check_timezone()

src/db_prep.py

@@ -0,0 +1,12 @@
+import os
+from dotenv import load_dotenv
+
+os.environ['RUN_TIMEZONE_CHECK'] = '0'
+
+from db_utils import init_db
+
+load_dotenv()
+
+if __name__ == "__main__":
+    print("Initializing database...")
+    init_db()

src/ingest.py

@@ -0,0 +1,19 @@
+import os
+import pandas as pd
+import pickle
+import minsearch
+
+DATA_PATH = os.getenv("DATA_PATH", "../dataset/data.csv")
+
+def load_index(data_path=DATA_PATH):
+    df = pd.read_csv(data_path)
+    documents = df.to_dict(orient="records")
+    print(documents[9])
+
+    index = minsearch.Index(
+        text_fields=['Questions', 'Answers'],
+        keyword_fields=["Question_ID"],
+    )
+
+    index.fit(documents)
+    return index

src/minsearch.py

@@ -0,0 +1,96 @@
+import pandas as pd
+
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.metrics.pairwise import cosine_similarity
+
+import numpy as np
+
+
+class Index:
+    """
+    A simple search index using TF-IDF and cosine similarity for text fields and exact matching for keyword fields.
+
+    Attributes:
+        text_fields (list): List of text field names to index.
+        keyword_fields (list): List of keyword field names to index.
+        vectorizers (dict): Dictionary of TfidfVectorizer instances for each text field.
+        keyword_df (pd.DataFrame): DataFrame containing keyword field data.
+        text_matrices (dict): Dictionary of TF-IDF matrices for each text field.
+        docs (list): List of documents indexed.
+    """
+
+    def __init__(self, text_fields, keyword_fields, vectorizer_params={}):
+        """
+        Initializes the Index with specified text and keyword fields.
+
+        Args:
+            text_fields (list): List of text field names to index.
+            keyword_fields (list): List of keyword field names to index.
+            vectorizer_params (dict): Optional parameters to pass to TfidfVectorizer.
+        """
+        self.text_fields = text_fields
+        self.keyword_fields = keyword_fields
+
+        self.vectorizers = {field: TfidfVectorizer(**vectorizer_params) for field in text_fields}
+        self.keyword_df = None
+        self.text_matrices = {}
+        self.docs = []
+
+    def fit(self, docs):
+        """
+        Fits the index with the provided documents.
+
+        Args:
+            docs (list of dict): List of documents to index. Each document is a dictionary.
+        """
+        self.docs = docs
+        keyword_data = {field: [] for field in self.keyword_fields}
+
+        for field in self.text_fields:
+            texts = [doc.get(field, '') for doc in docs]
+            self.text_matrices[field] = self.vectorizers[field].fit_transform(texts)
+
+        for doc in docs:
+            for field in self.keyword_fields:
+                keyword_data[field].append(doc.get(field, ''))
+
+        self.keyword_df = pd.DataFrame(keyword_data)
+
+        return self
+
+    def search(self, query, filter_dict={}, boost_dict={}, num_results=10):
+        """
+        Searches the index with the given query, filters, and boost parameters.
+
+        Args:
+            query (str): The search query string.
+            filter_dict (dict): Dictionary of keyword fields to filter by. Keys are field names and values are the values to filter by.
+            boost_dict (dict): Dictionary of boost scores for text fields. Keys are field names and values are the boost scores.
+            num_results (int): The number of top results to return. Defaults to 10.
+
+        Returns:
+            list of dict: List of documents matching the search criteria, ranked by relevance.
+        """
+        query_vecs = {field: self.vectorizers[field].transform([query]) for field in self.text_fields}
+        scores = np.zeros(len(self.docs))
+
+        # Compute cosine similarity for each text field and apply boost
+        for field, query_vec in query_vecs.items():
+            sim = cosine_similarity(query_vec, self.text_matrices[field]).flatten()
+            boost = boost_dict.get(field, 1)
+            scores += sim * boost
+
+        # Apply keyword filters
+        for field, value in filter_dict.items():
+            if field in self.keyword_fields:
+                mask = self.keyword_df[field] == value
+                scores = scores * mask.to_numpy()
+
+        # Use argpartition to get top num_results indices
+        top_indices = np.argpartition(scores, -num_results)[-num_results:]
+        top_indices = top_indices[np.argsort(-scores[top_indices])]
+
+        # Filter out zero-score results
+        top_docs = [self.docs[i] for i in top_indices if scores[i] > 0]
+
+        return top_docs

src/rag.py

@@ -0,0 +1,145 @@
+import json
+from time import time
+from groq import Groq
+from dotenv import load_dotenv
+import os
+import ingest
+import logging
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+load_dotenv()
+
+groq_api_key = os.getenv("GROQ_API_KEY")
+client = Groq(api_key=groq_api_key)
+
+# Load the search index
+try:
+    index = ingest.load_index()
+except Exception as e:
+    logger.error(f"Failed to load index: {e}")
+    raise
+
+if index is None:
+    raise ValueError("Search index could not be loaded")
+
+def search(query):
+    try:
+        results = index.search(
+            query=query,
+            num_results=10
+        )
+        return results
+    except Exception as e:
+        logger.error(f"Error in search function: {e}")
+        return []
+
+prompt_template = """ 
+You are an expert mental health assistant specialized in providing detailed and accurate answers based on the given context. Answer the QUESTION based on the CONTEXT from our mental health database. Use only the facts from the CONTEXT when answering the QUESTION.
+
+Here is the context:
+
+Context: {context}
+
+Please answer the following question based on the provided context:
+
+Question: {question}
+
+Provide a detailed and informative response. Ensure that your answer is clear, concise, and directly addresses the question while being relevant to the context provided.
+
+Your response should be in plain text and should not include any code blocks or extra formatting.
+
+Answer:
+""".strip()
+
+entry_template = """ 
+questions={Questions}
+answers={Answers}
+""".strip()
+
+def build_prompt(query, search_results):
+    context = ""
+    for doc in search_results:
+        context = context + entry_template.format(**doc) + "\n\n"
+    prompt = prompt_template.format(question=query, context=context).strip()
+    return prompt
+
+def llm(prompt, model="mixtral-8x7b-32768"):
+    start_time = time()
+    response = client.chat.completions.create(
+        model=model, messages=[{"role": "user", "content": prompt}]
+    )
+
+    answer = response.choices[0].message.content
+
+    token_stats = {
+        "prompt_tokens": response.usage.prompt_tokens,
+        "completion_tokens": response.usage.completion_tokens,
+        "total_tokens": response.usage.total_tokens,
+    }
+    end_time = time()
+    response_time = end_time - start_time
+
+    return answer, token_stats, response_time    
+
+def evaluate_relevance(question, answer, model='mixtral-8x7b-32768'):
+    eval_prompt = f"""
+You are an expert evaluator for a Retrieval-Augmented Generation (RAG) system.
+Your task is to analyze the relevance of the generated answer to the given question.
+Based on the relevance of the generated answer, you will classify it
+as "NON_RELEVANT", "PARTLY_RELEVANT", or "RELEVANT".
+
+Here is the data for evaluation:
+Question: {question}
+Answer: {answer}
+
+Please analyze the content and context of the generated answer in relation to the question
+and provide your evaluation in parsable JSON without using code blocks:
+
+"Relevance": "NON_RELEVANT" | "PARTLY_RELEVANT" | "RELEVANT",
+"Explanation": "[Provide a brief explanation for your evaluation]"
+""".strip()
+
+    evaluation, tokens, _ = llm(eval_prompt, model)
+    
+    try:
+        json_eval = json.loads(evaluation)
+        relevance = json_eval['Relevance'].upper()  # Ensure it's uppercase
+        if relevance not in ["NON_RELEVANT", "PARTLY_RELEVANT", "RELEVANT"]:
+            logger.warning(f"Unexpected relevance value: {relevance}. Defaulting to PARTLY_RELEVANT.")
+            relevance = "PARTLY_RELEVANT"
+        return relevance, json_eval['Explanation'], tokens
+    except json.JSONDecodeError:
+        logger.error(f"Failed to parse evaluation JSON: {evaluation}")
+        return "PARTLY_RELEVANT", "Failed to parse evaluation", tokens
+
+
+def rag(query, model="mixtral-8x7b-32768"):
+    t0 = time()
+
+    search_results = search(query)
+    prompt = build_prompt(query, search_results)
+    answer, tokens, response_time = llm(prompt, model=model)
+
+    relevance, explanation, eval_tokens = evaluate_relevance(query, answer, model=model)
+    
+    t1 = time()
+    took = t1 - t0
+
+    answer_data = {
+        'answer': answer,
+        'model_used': model,
+        'response_time': response_time,
+        'relevance': relevance,
+        'relevance_explanation': explanation,
+        'prompt_tokens': tokens['prompt_tokens'],
+        'completion_tokens': tokens['completion_tokens'],
+        'total_tokens': tokens['total_tokens'],
+        'eval_prompt_tokens': eval_tokens['prompt_tokens'],
+        'eval_completion_tokens': eval_tokens['completion_tokens'],
+        'eval_total_tokens': eval_tokens['total_tokens'],
+    }
+
+    return answer_data

src/synthetic_data.py

@@ -0,0 +1,114 @@
+import os
+import random
+from datetime import datetime, timedelta
+from zoneinfo import ZoneInfo
+import time
+import logging
+from db import save_conversation, save_feedback, get_db_connection
+import uuid
+
+# Set up logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+# Use the same timezone as in the original code
+TZ_INFO = os.getenv("TZ", "Europe/Berlin")
+tz = ZoneInfo(TZ_INFO)
+
+# Sample data for generating synthetic conversations
+QUESTIONS = [
+    "How can I manage stress?",
+    "What are the symptoms of depression?",
+    "How to improve sleep quality?",
+    "What is cognitive behavioral therapy?",
+    "How to deal with anxiety attacks?",
+]
+
+MODELS = ["gemma2-9b-it", "llama-3.1-70b-versatile", "llama3-70b-8192", "mixtral-8x7b-32768"]
+RELEVANCE_OPTIONS = ["NON_RELEVANT", "PARTLY_RELEVANT", "RELEVANT"]
+
+def generate_answer(question):
+    return f"Here's some information about {question.lower()[:-1]}: ..."
+
+def generate_conversation(timestamp):
+    question = random.choice(QUESTIONS)
+    model = random.choice(MODELS)
+    conversation_id = str(uuid.uuid4())
+    
+    answer_data = {
+        "answer": generate_answer(question),
+        "model_used": model,
+        "response_time": round(random.uniform(0.5, 3.0), 2),
+        "relevance": random.choice(RELEVANCE_OPTIONS),
+        "relevance_explanation": "This is a synthetic relevance explanation.",
+        "prompt_tokens": random.randint(50, 200),
+        "completion_tokens": random.randint(100, 500),
+        "total_tokens": random.randint(150, 700),
+        "eval_prompt_tokens": random.randint(20, 100),
+        "eval_completion_tokens": random.randint(50, 200),
+        "eval_total_tokens": random.randint(70, 300),
+    }
+    
+    return conversation_id, question, answer_data, timestamp
+
+def generate_feedback(conversation_id, timestamp):
+    feedback = random.choice([-1, 1])
+    return conversation_id, feedback, timestamp
+
+def populate_historical_data():
+    end_time = datetime.now(tz)
+    start_time = end_time - timedelta(days=7)  # Generate a week's worth of data
+    current_time = start_time
+
+    while current_time <= end_time:
+        conversation_id, question, answer_data, timestamp = generate_conversation(current_time)
+        try:
+            save_conversation(conversation_id, question, answer_data, timestamp)
+            logger.info(f"Saved historical conversation: {conversation_id} at {timestamp}")
+            
+            # Generate feedback for some conversations
+            if random.random() < 0.7:  # 70% chance of feedback
+                feedback_timestamp = timestamp + timedelta(minutes=random.randint(1, 10))
+                save_feedback(*generate_feedback(conversation_id, feedback_timestamp))
+                logger.info(f"Saved historical feedback for conversation: {conversation_id}")
+        except Exception as e:
+            logger.error(f"Error saving historical data: {e}")
+        
+        current_time += timedelta(minutes=random.randint(5, 60))
+
+def generate_live_data():
+    while True:
+        timestamp = datetime.now(tz)
+        conversation_id, question, answer_data, _ = generate_conversation(timestamp)
+        try:
+            save_conversation(conversation_id, question, answer_data, timestamp)
+            logger.info(f"Generated live conversation: {conversation_id} at {timestamp}")
+
+            # Generate feedback for some conversations
+            if random.random() < 0.7:  # 70% chance of feedback
+                feedback_timestamp = timestamp + timedelta(seconds=random.randint(30, 300))
+                save_feedback(*generate_feedback(conversation_id, feedback_timestamp))
+                logger.info(f"Generated live feedback for conversation: {conversation_id}")
+        except Exception as e:
+            logger.error(f"Error saving live data: {e}")
+
+        time.sleep(random.uniform(1, 5))  # Wait for 1-5 seconds before generating the next data point
+
+if __name__ == "__main__":
+    logger.info("Starting synthetic data generation")
+    
+    # Ensure database connection is working
+    try:
+        conn = get_db_connection()
+        conn.close()
+        logger.info("Database connection successful")
+    except Exception as e:
+        logger.error(f"Failed to connect to the database: {e}")
+        exit(1)
+
+    logger.info("Populating historical data...")
+    populate_historical_data()
+    logger.info("Historical data population complete")
+
+    logger.info("Starting live data generation...")
+    generate_live_data()